Autonomous vehicle localization using passive image data

ABSTRACT

A system to use submaps to control operation of a vehicle is disclosed. A storage system may be provided with a vehicle to store a collection of submaps that represent a geographic area where the vehicle may be driven. A programmatic interface may be provided to receive submaps and submap updates independently of other submaps.

RELATED APPLICATIONS

This application claims benefit of priority to Provisional U.S. PatentApplication No. 62/412,041, filed on Oct. 24, 2016; and to ProvisionalU.S. Patent Application No. 62/357,903, filed on Jul. 1, 2016; each ofthe aforementioned priority applications being hereby incorporated byreference in their respective entirety.

TECHNICAL FIELD

Examples described herein relate to a submap system for autonomouslyoperating vehicles.

BACKGROUND

Vehicles are increasingly implementing autonomous control. Manyhuman-driven vehicles, for example, have modes in which the vehicle canfollow in a lane and change lanes.

Fully autonomous vehicles refer to vehicles which can replace humandrivers with sensors and computer-implemented intelligence, sensors andother automation technology. Under existing technology, autonomousvehicles can readily handle driving with other vehicles on roadways suchas highways.

Autonomous vehicles, whether human-driven hybrids or fully autonomous,operate using data that provides a machine understanding of theirsurrounding area.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an example map system for enabling autonomous controland operation of a vehicle.

FIG. 2 illustrates a submap network service, according to one or moreembodiments.

FIG. 3 illustrates a submap data aggregation that stores and linksmultiple versions of submaps, collectively representing linked roadwaysegments for a geographic region, according to one or more examples.

FIG. 4 illustrates an example of a control system for an autonomousvehicle.

FIG. 5 is a block diagram of a vehicle system on which an autonomousvehicle control system may be implemented.

FIG. 6 is a block diagram of a network service or computer system onwhich some embodiments may be implemented.

FIG. 7 illustrates an example method for operating a vehicle using asubmap system, according to one or more examples.

FIG. 8 illustrates an example method for distributing mappinginformation to vehicles of a geographic region for use in autonomousdriving, according to one or more examples.

FIG. 9 illustrates an example method for providing guidance toautonomous vehicles.

FIG. 10 illustrates an example sensor processing sub-system for anautonomous vehicle, according to one or more embodiments.

FIG. 11 illustrates an example of a vehicle on which an example of FIG.10 is implemented.

FIG. 12 illustrates an example method for determining a location of avehicle in motion using vehicle sensor data, according to an embodiment.

FIG. 13 illustrates a method for determining a location of a vehicle inmotion using image data captured by the vehicle, according to anembodiment.

FIG. 14 illustrates a method for determining a location of a vehicle inmotion using an image point cloud and image data captured by thevehicle, according to an embodiment.

FIG. 15 illustrates an example method in which the perception output isused by a vehicle to process a scene.

DETAILED DESCRIPTION

Examples herein describe a system to use submaps to control operation ofa vehicle. A storage system may be provided with a vehicle to store acollection of submaps that represent a geographic area where the vehiclemay be driven. A programmatic interface may be provided to receivesubmaps and submap updates independently of other submaps.

As referred to herein, a submap is a map-based data structure thatrepresents a geographic area of a road segment, with data sets that arecomputer-readable to facilitate autonomous control and operation of avehicle. In some examples, a submap may include different types of datacomponents that collectively provide a vehicle with information that isdescriptive of a corresponding road segment. In some examples, a submapcan include data that enables a vehicle to traverse a given road segmentin a manner that is predictive or responsive to events which canotherwise result in collisions, or otherwise affect the safety of peopleor property. Still further, in some examples, a submap provides a datastructure that can carry one or more data layers which fulfill a dataconsumption requirement of a vehicle when the vehicle is autonomouslynavigated through an area of a road segment. The data layers of thesubmap can include, or may be based on, sensor information collectedfrom a same or different vehicle (or other source) which passed throughthe same area on one or more prior instances.

One or more embodiments described herein provide that methods,techniques, and actions performed by a computing device are performedprogrammatically, or as a computer-implemented method. Programmatically,as used herein, means through the use of code or computer-executableinstructions. These instructions can be stored in one or more memoryresources of the computing device. A programmatically performed step mayor may not be automatic.

One or more embodiments described herein can be implemented usingprogrammatic modules, engines, or components. A programmatic module,engine, or component can include a program, a sub-routine, a portion ofa program, or a software component or a hardware component capable ofperforming one or more stated tasks or functions. As used herein, amodule or component can exist on a hardware component independently ofother modules or components. Alternatively, a module or component can bea shared element or process of other modules, programs or machines.

Numerous examples are referenced herein in context of an autonomousvehicle. An autonomous vehicle refers to any vehicle which is operatedin a state of automation with respect to steering and propulsion.Different levels of autonomy may exist with respect to autonomousvehicles. For example, some vehicles today enable automation in limitedscenarios, such as on highways, provided that drivers are present in thevehicle. More advanced autonomous vehicles drive without any humandriver inside the vehicle. Such vehicles often are required to makeadvance determinations regarding how the vehicle is behave givenchallenging surroundings of the vehicle environment.

Map System

FIG. 1 illustrates an example map system for enabling autonomous controland operation of a vehicle. In an example of FIG. 1, a submapinformation processing system (“SIPS 100”) may utilize submaps whichindividually represent a corresponding road segment of a road network.By way of example, each submap can represent a segment of a roadway thatmay encompass a block, or a number of city blocks (e.g., 2-5 cityblocks). Each submap may carry multiple types of data sets, representingknown information and attributes of an area surrounding thecorresponding road segment. The SIPS 100 may be implemented as part of acontrol system for a vehicle 10 that is capable of autonomous driving.In this way, the SIPS 100 can be implemented to enable the vehicle 10,operating under autonomous control, to obtain known attributes andinformation for an area of a road segment. The known attributes andinformation, which are additive to the identification of the roadnetwork within the submap, enable the vehicle 10 to responsively andsafely navigate through the corresponding road segment.

Among other utilities, the SIPS 100 can provide input for an autonomousvehicle control system 400 (see FIG. 4), in order to enable the vehicle10 to operate and (i) plan/implement a trajectory or route through aroad segment based on prior knowledge about the road segment, (ii)process sensor input about the surrounding area of the vehicle withunderstanding about what types of objects are present, (iii) detectevents which can result in potential harm to the vehicle, or persons inthe area, and/or (iv) detect and record conditions which can affectother vehicles (autonomous or not) passing through the same roadsegment. In variations, other types of functionality can also beimplemented with use of submaps. For example, in some variations,individual submaps can also carry data for enabling the vehicle 10 todrive under different driving conditions (e.g., weather variations, timeof day variations, traffic variations, etc.).

In some examples, the vehicle 10 can locally store a collection ofstored submaps 105 which are relevant to a geographic region that thevehicle 10 is anticipated to traverse during a given time period (e.g.,later in trip, following day, etc.). The collection of stored submaps105 may be retrieved from, for example, a submap network service 200(see FIG. 2) that maintains and updates a larger library of submaps formultiple vehicles (or user-vehicles).

With respect to the vehicle 10, each of the stored submaps 105 canrepresent an area of a road network, corresponding to a segment of theroad network and its surrounding area. As described with some examples,individual submaps may include a collection of data sets that representan area of the road segment within a geographic region (e.g., city, orportion thereof). Furthermore, each of the submap 105 can include datasets (sometimes referred to as data layers) to enable an autonomousvehicle 10 to perform operations such as localization, as well asdetection and recognition of dynamic objects.

In an example of FIG. 1, the SIPS 100 includes a submap retrievalcomponent 110, a submap processing component 120, a submap networkinterface 130, a submap manager 136, and roadway data aggregationprocesses 140. As the SIPS 100 may be implemented as part of the AVcontrol system 400, the SIPS 100 may utilize or incorporate resources ofthe vehicle 10, including processing and memory resources, as well assensor devices of the vehicle (e.g., Lidar, stereoscopic and/or depthcameras, video feed sonar, radar, etc.). In some examples, the SIPS 100employs the submap network interface 130, in connection with the submapnetwork service 200 (FIG. 2), to receive new or replacement submaps 131and/or submap updates 133. In some examples, the submap networkinterface 130 can utilize one or more wireless communication interfaces94 of the vehicle 10 in order to wireless communicate with the submapnetwork service 200 (e.g., see FIG. 2) and receive new or replacementsubmaps 131 and/or submap updates 133. In variations, the submap networkinterface 130 can receive new or replacement submaps 131 and/or submapupdates 133 from other remote sources, such as other vehicles.

In addition to receiving the new or replacement submaps 131 and submapupdates 133, the submap network interface 130 can communicate vehicledata 111 to the submap network service 200. The vehicle data 111 caninclude, for example, the vehicle location and/or vehicle identifier.

The submap manager 136 can receive the new or replacement submaps 131and/or submap updates 133, and create a stored collection of submaps 105utilizing an appropriate memory component 104A, 104B. In some examples,the submaps have a relatively large data size, and the vehicle 10retrieves the new submaps 131 when such submaps are needed. The submapnetwork interface 130 can also receive submap updates 133 for individualsubmaps, or groups of submaps, stored as part of the collection 105. Thesubmap manager 136 can include processes to manage the storage,retrieval and/or updating of stored submaps 105, in connection with, forexample, the submap network service 200 (see FIG. 2) and/or other submapdata sources (e.g., other vehicles).

In some examples, the submap manager 136 can implement co-locationstorage operations 109 as a mechanism to manage the stored submaps ofthe collection 105 in a manner that enables the data sets of the submapsto be rapidly retrieved and utilized by an AV control system 400. Insome examples, the individual submaps of the collection 105 may includea combination of rich data sets which are linked by other data elements(e.g., metadata). An example submap with organized data layers isprovided with FIG. 3. Given the range in velocity of vehicle 10, and theamount of data which is collected and processed through the varioussensors of the vehicle 10, examples recognize that storing the data setsof individual submaps in physical proximity to one another on the memorycomponents 104A, 104B of the vehicle 10 can reduce memory managementcomplexity and time lag when individual submaps of the collection 105are locally retrieved and utilized. Examples further recognize thatphysically grouping individually stored submaps 105, representingadjacent or proximate geographic areas in physical proximity to oneanother, on respective memory components 104A, 104B of the vehicle 10further promotes the ability of the SIPS 100 to make timely transitionsfrom one submap to another.

In the example shown by FIG. 1, the SIPS 100 utilizes multiple memorycomponents 104A, 104B (collectively “memory components 104”). The submapmanager 136 can implement co-location storage operations 109 to storesubmaps 105 relating to a particular area or sub-region of a roadnetwork on only one of the memory components 104. In variations, thesubmap manager 136 can implement co-location storage operations 109 toidentify memory cells of the selected memory component 104 which areadjacent or near one another for purpose of carrying data of a givensubmap, or data for two or more adjacent submaps.

According to some examples, the submap retrieval component 110 includesprocesses for performing a local search or retrieval for stored submaps105 provided with the memory components 104. The submap retrievalcomponent 110 can signal submap selection input 123 to the submapmanager 136 in order to locally retrieve 107 one or more submaps 125 forimmediate processing (e.g., sub-region for upcoming segment of trip). Insome instances, examples provide that the selection input 123 can begenerated from a source that provides an approximate location of thevehicle 10. In one implementation, the selection input 123 is used toretrieve an initial set of submaps 125 for a road trip of the vehicle10. The selection input 123 may be obtained from, for example, the lastknown location of the vehicle prior to the vehicle being turned off inthe prior use. In other variations, the selection input 123 can beobtained from a location determination component (e.g., a satellitenavigation component, such as provided by a Global Navigation SatelliteSystem (or “GNSS”) type receiver) of the vehicle 10.

The submap manager 136 may respond to receiving the selection input 123by accessing a database of the local memory components 104 where arelevant portion of the collection of submaps 105 are stored. The submapmanager 136 may be responsive to the selection input 123, in order toretrieve from the local memory components 104 an initial set of submaps125. The initial set of submaps 125 can include one or multiple submaps,each of which span a different segment of a road or road network thatincludes, for example, a geographic location corresponding to theselection input 123.

Each of the stored submaps 105 may include data layers corresponding tomultiple types of information about a corresponding road segment. Forexample, submaps may include data to enable the SIPS 100 to generate apoint cloud of its environment, with individual points of the cloudproviding information about a specific point in three-dimensional spaceof the surrounding environment. In some examples, the individual pointsof the point cloud may include or be associated with image data thatvisually depict a corresponding point in three-dimensional space. Imagedata which forms individual points of a point cloud are referred to as“imagelets”. In some examples, the imagelets of a point cloud may depictsurface elements, captured through Lidar (sometimes referred to as“surfels”). Still further, in some examples, the imagelets of a pointcloud may include other information, such as a surface normal (or unitvector describing orientation). As an addition or variation, the pointsof the point cloud may also be associated with other types ofinformation, including semantic labels, road network information, and/ora ground layer data set. In some examples, each of the stored submaps105 may include a feature set 113 that identifies features which arepresent in a surrounding area of the road segment corresponding to thatsubmap.

The submap processing component 120 may include submap start logic 114for scanning individual submaps of an initially retrieved submap set125, to identify the likely submap for an initial location of thevehicle 10. In one implementation, the submap processing component 120implements the start component 122 as a coarse or first-pass process tocompare the submap feature set 113 of an initially retrieved submapagainst a current sensor state 493, as determined from one or moresensor interfaces or components of the vehicle's sensor system 492. Thestart logic 114 may perform the comparison to identify, for example, acurrent submap 145 of the initial set which contains the feature of alandmark detected as being present in the current sensor state 493 ofthe vehicle 10. Once the current submap 145 is identified, the submapprocessing component 120 can perform a more refined localization processusing the current submap 145, in order to determine a more preciselocation of the vehicle 10 relative to the starting submap. In someexamples, the submap processing component 120 can track the movement ofthe vehicle 10 in order to coordinate the retrieval and/or processing ofa next submap that is to be the current submap 145, corresponding to anadjacent road segment that the vehicle traverses on during a trip.

With further reference to an example of FIG. 1, the submap retrievalcomponent 110 can select the current submap 145 for processing by thesubmap processing component 120. The submap processing component 120 canprocess the current submap 145 contemporaneously, or nearcontemporaneously, with the vehicle's traversal of the correspondingroad segment. The data layers provided with the current submap 145enable the vehicle 10 to drive through the road segment in a manner thatis predictive or responsive to events or conditions which are otherwiseunknown.

According to some examples, the submap retrieval and processingcomponents 110, 120 can execute to retrieve and process a series ofsubmaps 125 in order to traverse a portion of a road network thatencompasses multiple road segments. In this manner, each submap of theseries can be processed as the current submap 145 contemporaneously withthe vehicle 10 passing through the corresponding area or road segment.In some examples, the submap processing component 120 extracts, orotherwise determines the submap feature set 113 for an area of the roadnetwork that the vehicle traverses. The submap processing component 120compares the submap feature set 113 of the current submap 145 to thecurrent sensor state 493 as provided by the vehicle's sensor system 492.The comparison can involve, for example, performing transformations ofsensor data, and/or image processing steps such as classifying and/orrecognizing detected objects or portions of a scene.

As the vehicle progresses on a trip, some examples provide for thesubmap processing component 120 to use tracking logic 118 to maintain anapproximate position of the vehicle 10 until localization is performed.The tracking logic 118 can process, for example, telemetry information(e.g., accelerometer, speedometer) of the vehicle, as well as follow onsensor data from the sensor system 492 of the vehicle, to approximatethe progression and/or location of the vehicle as it passes through agiven area of a submap. The tracking logic 118 can trigger and/orconfirm the progression of the vehicle from, for example, one submap toanother, or from one location within a submap to another location of thesame submap. After a given duration of time, the submap processingcomponent 120 can process a next submap contemporaneously with thevehicle's progression into the area represented by the next submap.

In some examples, the submap processing component 120 processes thecurrent submap 145 to determine outputs for use with different logicalelements of the AV control system 400. In one implementation, the outputincludes localization output 121, which can identify a precise or highlygranular location of the vehicle, as well as the pose of the vehicle. Insome examples, the location of the vehicle can be determined to a degreethat is more granular than that which can be determined from, forexample, a satellite navigation component. As an addition or variation,the output of the submap processing component 120 includes object datasets, which locate and label a set of objects detected from thecomparison of the current sensor state 493 and the submap feature set113.

According to some examples, the submap processing component 120 caninclude localization component 122 to perform operations for determiningthe localization output. The localization output 121 can be determinedat discrete instances while the vehicle 10 traverses the area of theroad segment corresponding to the current submap 145. The localizationoutput 121 can include location coordinate 117 and pose 119 of thevehicle 10 relative to the current submap 145. In some examples, thelocalization component 122 can compare information from the currentsensor state 493 (e.g., Lidar data, imagery, sonar, radar, etc.) to thefeature set 113 of the current submap 145. Through sensor datacomparison, the location of the vehicle 10 can be determined withspecificity that is significantly more granular than what can bedetermined through use of a satellite navigation component. In someexamples, the location coordinates 117 can specify a position of thevehicle within the reference frame of the current submap 145 to be of amagnitude that is less than 1 foot (e.g., 6 inches or even less,approximate diameter of a tire, etc.). In this way, the locationcoordinates 117 can pinpoint the position of the vehicle 10 bothlaterally and in the direction of travel. For example, for a vehicle inmotion, the location coordinates 117 can identify any one or more of:(i) the specific lane the vehicle occupies, (ii) the position of thevehicle within an occupied lane (e.g., on far left side of a lane) ofthe vehicle, (iii) the location of the vehicle in between lanes, and/or(iv) a distance of the vehicle from a roadside boundary, such as ashoulder, sidewalk curb or parked car.

As an addition or a variation, the submap processing component 120 caninclude perception component 124 which provides perception output 129representing objects that are detected (through analysis of the currentsensor state 493) as being present in the area of the road network. Theperception component 124 can determine the perception output to include,for example, a set of objects (e.g., dynamic objects, road features,etc.). In determining the perception output 129, the perceptioncomponent 124 can compare detected objects from the current sensor state493 with known and static objects identified with the submap feature set113. The perception component 124 can generate the perception output 129to identify (i) static objects which may be in the field of view, (ii)non-static objects which may be identified or tracked, (iii) an imagerepresentation of the area surrounding a vehicle with static objectsremoved or minimized, so that the remaining data of the current sensorstate 493 is centric to dynamic objects.

In order to navigate the vehicle on a trip, the submap retrievalcomponent 110 identifies and retrieves next submap(s) from the submapmanager 136. The next submaps that are retrieved by the submap retrievalcomponent 110 can be identified from, for example, a determinedtrajectory of the vehicle 10 and/or a planned or likely route of thevehicle 10. In this way, the submap retrieval component 110 canrepeatedly process, during a given trip, a series of submaps to reflecta route of the vehicle over a corresponding portion of the road network.The submap processing component 120 can process the current submaps 125from each retrieved set in order to determine localization output 121and perception output 129 for the AV control system 400.

According to some examples, the stored submaps 105 can be individuallyupdated, independently of other submaps of a geographic region. As aresult, the SIPS 100 can manage updates to its representation of ageographic region using smaller and more manageable units of targetdata. For example, when conditions or events to a specific segment ofthe road network merit an update, the SIPS 100 can receive and implementupdates to a finite set of submaps (e.g., one to three submaps, squarekilometer or half-kilometer, etc.) rather than update a maprepresentation for the larger geographic region. Additionally, theability for the submap processing component 120 to use submaps which areindependently updated allows for the vehicle 10 and/or other vehicles ofthe geographic region to aggregate information for enabling updates tosubmaps used on other vehicles.

As described with other examples, the vehicle 10 can operate as part ofa group of vehicles (or user-vehicles) which utilize submaps in order toautonomously navigate through a geographic region. In cases wheremultiple vehicles using submaps traverse the road network of a givengeographic region, some embodiments provide that individual vehicles canoperate as observers for conditions and patterns from which submapfeatures can be determined. As described with other examples, the submapnetwork service 200 (FIG. 2) can implement a variety of processes inorder to generate sensor data, labels, point cloud information and/orother data from which submap data can be generated and used to updatecorresponding submaps. For a given geographic region, different submapscan be updated based on events, changing environmental conditions (e.g.weather) and/or refinements to existing models or submap feature sets.

With operation of the vehicle 10, the roadway data aggregation processes140 can receive and aggregate sensor data input 143 from one or morevehicle sensor sources (shown collectively as vehicle sensor system492). The sensor data input 143 can, for example, originate in raw orprocessed form from sensors of the vehicle 10, or alternatively, fromsensor components of the vehicle which process the raw sensor data. Theroadway data aggregation processes 140 can process and aggregate thesensor data input 143 to generate aggregated sensor data 141. Theaggregated sensor data 141 may be generated in accordance with aprotocol, which can specify raw data processing steps (e.g., filtering,refinements), data aggregation, conditions for synchronous orasynchronous (e.g., offline) transmissions of aggregated sensor data 141and/or other aspects of sensor data aggregation, storage, andtransmission.

In one implementation, the aggregate sensor data 141 can be transmittedto the submap network service 200 via the wireless communicationinterface 94 of the vehicle 10. In variations, the aggregate sensor data141 can be used to generate local updates for one or more stored submaps105. In some variations, the roadway data aggregation processes 140 cancollect sensor data input 143, and perform, for example, varianceanalysis using variance logic 144. The variance logic 144 may be used togenerate a local submap update 149, which can be used to update acorresponding submap of the collection 105 via the submap manager 136.

While examples provide for submaps to be independently updated, examplesfurther recognize that updates to submaps can make the use of suchsubmaps incompatible with other submaps. For example, if one submap isof a given area is updated while an adjacent submap is not, then thesubmap processing component 120 may not able to transition from onesubmap to the adjacent submap. By way of example, the update for thesubmap may cause the submap processing component 120 to process thesubmap using an algorithm or logic that is different than what waspreviously used. In some examples, the submap processing component 120can be updated in connection with updates to submaps that are receivedand processed on that vehicle. For example, new submaps 131 received bythe submap network interface 130 may include instructions, code, ortriggers that are executable by the SIPS 100 (e.g., by the submapmanager 136) to cause the vehicle 10 to retrieve or implement aparticular logic from which the submap is subsequently processed.

According to some examples, the new submaps 131 retrieved from thesubmap network service 200 (or other remote source) are versioned toreflect what submap updates are present on the particular submap. Insome examples, an update to a given submap can affect a particular typeof data set or data layer on the submap. Still further, in othervariations, the update to the submap can be programmatic (e.g., alter analgorithm used to process a data layer of the submap) or specific todata sets used by processes which consume the data layer of the submap.

Still further, in some variations, the submap retrieval component 110may include versioning logic 116 which identifies the version of thesubmap (e.g., from the UID of the submap) and then retrieves a nextsubmap that is of the same or compatible version. As described withother examples, the new submaps 131 of the collection can be structuredto include connector data sets 308 (see FIG. 3) which enables thevehicle to stitch consecutive submaps together as the vehicle 10progresses through a road network.

Submap Network Service

FIG. 2 illustrates a submap network service, according to one or moreembodiments. In one implementation, the submap network service 200 canbe implemented on a server, or combination of servers, which communicatewith network enabled vehicles that traverse a road network of ageographic region. In a variation, the submap network service 200 can beimplemented in alternative computing environments, such as a distributedenvironment. For example, some or all of the functionality described mayimplemented on a vehicle, or combination of vehicles, which collectivelyform a mesh or peer network. In some examples, a group of vehicles, inoperation within a geographic region, may implement a mesh network, orpeer-to-peer network, to transmit and receive data, including submapsand data for creating or updating submaps.

In an example of FIG. 2, the submap network service 200 includes avehicle interface 210, a vehicle monitor 220, a sensor data analysissub-system 230 and a submap service manager 240. The vehicle interface210 provides the network interface that can communicate with one ormultiple vehicles in a given geographic region. In some implementations,the vehicle interface 210 receives communications, which include vehicledata 211, from individual vehicles that wirelessly communicate with thesubmap network service 200 during their respective operations. Thevehicle monitor 220 can receive, store and manage various forms ofvehicle data 211, including a vehicle identifier 213, a vehicle location215, and a current submap version 217 for each vehicle. The vehicle data211 can be stored in, for example, vehicle database 225.

According to some examples, the vehicle monitor 220 manages thetransmission of new submaps 231 and submap updates 233 to vehicles ofthe geographic region. The vehicle monitor 220 retrieves a set ofsubmaps 237 for individual vehicles 10 from the submap service manager240. In one implementation, the vehicle monitor 220 retrieve separatesets of submaps 237 for different vehicles, based on the vehicle data211 stored in the vehicle database 225. For example, the vehicle monitor220 can retrieve a submap set 237 for a given vehicle using the vehicleidentifier 213, the vehicle location 215 associated with the vehicleidentifier, and/or the current submap version 217 for the vehicleidentifier.

The submap service manager 240 can manage storage and retrieval ofindividual submaps 239 from a submap database 248. One or multiplesubmap sources can create submaps and/or update individual submap storedin the submap database 248 or similar memory structure. The submapservice manager 240 can include a submap selection and version matchingcomponent 242 that can select sets of submaps 237 for individualvehicles of a geographic region. The submap selection/version matchingcomponent 242 can select sets of submaps 237 for individual vehicles,based on the vehicle location 215 and the vehicle submap version 217. Inresponse to receiving the vehicle location 215, for example, the submapselection/version matching component 242 may search the submap database248 to identify submaps for the geographic region of the vehicle, havinga same or compatible submap version.

To retrieve a submap for a vehicle 10, the vehicle monitor 220 maycommunicate selection input 235 (based on or corresponding to thevehicle location 215) for the vehicle, as well as other informationwhich would enable the submap service manager 240 to select the correctsubmap and version for the vehicle at the given location. For example,the vehicle database 225 can associate a vehicle identifier with thesubmap version of the vehicle 10. In variations, the vehicle 10 cancommunicate its submap version when requesting submaps from the submapnetwork service 200.

The submap service manager 240 can initiate return of a new set ofsubmaps 237 for a given submap request of a vehicle. The new submap sets237 can be selected from the submap database 248 and communicated to aspecific vehicle via the vehicle interface 210. For example, individualvehicles 10 may carry (e.g., locally store and use) a limited set ofsubmaps, such as submaps which the vehicle is likely to need over agiven duration of time. But if the vehicle traverses the geographicregion such that submaps for other localities are needed, the vehicle 10may request additional submaps from the submap network service 200, andthen receive the new submap sets 237 based on the vehicle's potentialrange.

In some variations, the submap network service 200 also generates submapupdates 233 for vehicles of the geographic region. The submap updates233 for a given submap may correspond to any one of an updated submap,an updated data layer or component of the submap, or a data differentialrepresenting the update to a particular submap. As described in greaterdetail, the submap update 233 to a given submap may result in a newsubmap version.

According to some examples, the submap network service 200 can includesubmap distribution logic 246 to interface with the submap database 248.The submap distribution logic 246 may receive update signals 249signifying when new submap sets 237 and/or submap updates 233 aregenerated. The submap distribution logic 246 can trigger the vehiclemonitor 220 to retrieve new submap sets 237 and/or submap updates 233from the submap database 248 based on distribution input 245communicated from the submap distribution logic 246. The distributioninput 245 can identify vehicles by vehicle identifier 213, by class(e.g., vehicles which last received updates more than one week prior) orother designation. The vehicle monitor 220 can determine the selectioninput 235 for a vehicle or set of vehicles based on the distributioninput 245. The submap distribution logic 246 can generate thedistribution input 245 to optimize distribution of updates to individualsubmaps of the submap database 248. The submap distribution logic 246may also interface with the vehicle database 225 in order to determinewhich vehicles should receive new submap sets 237 and/or submap updates233 based on the vehicle identifier 213, vehicle location 215 and submapversion 217 associated with each vehicle. In this way, the submapdistribution logic 246 can cause the distribution of new submap sets 237and/or submap updates 233 to multiple vehicles of a given geographicregion in parallel, so that multiple vehicles can receive new submapsets 237 and/or submap updates 233 according to a priority distributionthat is optimized for one or more optimization parameters 247.

In one implementation, optimization parameter 247 can correspond to aproximity parameter that reflects a distance between a current locationof a vehicle and an area of the road network where submaps (of differentversions) have recently been updated. By way of example, the submapdistribution logic 246 can utilize the optimization parameter 247 toselect vehicles (or classes of vehicles) from the geographic regionwhich is prioritized to receive updated submaps. For example, vehicleswhich receive the series of submap sets 237 and updates 233 can includevehicles that are expected to traverse the regions of the road networkwhich have corresponding submap updates sooner, based on their proximityor historical pattern of use.

In variations, the optimization parameter 247 can also de-select orde-prioritize vehicles which, for example, may be too close to an areaof the geographic region that corresponds to the new submap sets 237 orsubmap updates. For vehicles that are too close, the de-prioritizationmay ensure the corresponding vehicle has time to receive and implementan updated submap before the vehicle enters a corresponding area of aroad network.

In variations, the optimization parameters 247 for determining selectionand/or priority of vehicles receiving new submap sets 237 and/or submapupdates 233. Still further, the submap distribution logic 246 canutilize the vehicle operational state to determine whether other updatesare to be distributed to the vehicle. For example, larger updates (e.g.,a relatively large number of new submaps) may require vehicles to benon-operational when the update is received and implemented. Thus, someexamples contemplate that at least some new submap sets 237 and/orsubmap updates 233 can be relatively small, and received and implementedby vehicles which are in an operational state (e.g., vehicles on trip).Likewise, some examples contemplate that larger updates can be deliveredto vehicles when those vehicles are in an off-state (e.g., in alow-power state by which updates can be received and implemented on thevehicle).

According to some examples, the submap network service 200 includesprocesses that aggregate sensor information from the vehicles thatutilize the submaps, in order to determine at least some updates tosubmaps in use. As an addition or variation, the submap network service200 can also receive and aggregate submap information from othersources, such as human driven vehicles, specialized sensor vehicles(whether human operated or autonomous), and/or network services (e.g.,pot hole report on Internet site). The sensor data analysis 230represents logic and processes for analyzing vehicle sensor data 243obtained from vehicles that traverse road segments represented by thesubmaps of the submap database 248. With reference to FIG. 1, forexample, the vehicle sensor data 243 can correspond to output of theroadway data aggregation process 140.

The sensor analysis sub-system 230 can implement processes and logic toanalyze the vehicle sensor data 243, and to detect road conditions whichcan or should be reflected in one or more data layers of a correspondingsubmap. Additionally, the sensor analysis sub-system 230 can generatedata sets as sensor analysis determinations 265 to modify and update thedata layers of the submaps to more accurately reflect a current orrecent condition of the corresponding road segment.

According to some examples, the sensor analysis sub-system 230implements sensor analysis processes on vehicle sensor data 243 (e.g.,three-dimensional depth image, stereoscopic image, video, Lidar, etc.).In one example, the sensor analysis sub-system 230 may include aclassifier 232 to detect and classify objects from the vehicle sensordata 243. Additionally, the sensor analysis sub-system 230 may includean image recognition process 234 to recognize features from the sensordata for the detected objects. The classifier 232 and the imagerecognition process 234 can generate sensor analysis determinations 265.The sensor analysis determinations 265 can specify a classification ofthe detected objects, as well as features of the classified object.

Other types of sensor analysis processes may also be used. According tosome examples, the sensor analysis sub-system 230 includes patternanalysis component 236 which implements pattern analysis on aggregationsof sensor analysis determinations 265 for a particular road segment orarea. In some examples, the vehicle data 211 links the vehicle sensordata 243 to one or more localization coordinate 117 (see FIG. 1), sothat the vehicle sensor data 243 is associated with a precise location.The pattern analysis component 236 can process the vehicle sensor data243 of multiple vehicles, for a common area (as which may be defined bythe localization coordinate 117 communicated by the individual vehicles)and over a defined duration of time (e.g., specific hours of a day,specific days of a week, etc.). The sensor analysis determinations 265can be aggregated, and used to train models that are capable ofrecognizing objects in sensor data, particularly with respect togeographic regions and/or lighting conditions. Still further, theprocesses of the sensor analysis sub-system 230 can be aggregated todetect temporal or transient conditions, such as time of day whentraffic conditions arise. As described with some other examples, thesensor analysis determinations 265 can include object detectionregarding the formation of instantaneous road conditions (e.g., new roadhazard), as well as pattern detection regarding traffic behavior (e.g.,lane formation, turn restrictions in traffic intersections, etc.).

Accordingly, the sensor analysis determinations 265 of the sensoranalysis sub-system 230 can include classified objects, recognizedfeatures, and traffic patterns. In order to optimize analysis, somevariations utilize the feature set 223 of a corresponding submap for anarea of a road network that is under analysis.

In some examples, a baseline component 252 can extract baseline input257 from the submap of an area from which aggregated sensor analysisdata is being analyzed. The baseline input 257 can include or correspondto the feature set 223 of the submaps associated with the area of theroad network. The baseline component 252 can extract, or otherwiseidentify the baseline input 257 as, for example, a coarse depiction ofthe road surface, static objects and/or landmarks of the area of thesubmap. The baseline input 257 can provide a basis for the sensor dataanalysis 230 to perform classification and recognition, and to identifynew and noteworthy objects and conditions.

As an addition or variation, the submap comparison component 250includes processes and logic which can compare sensor analysisdeterminations 265 of the sensor analysis sub-system 230, with thefeature sets 223 of the corresponding submaps for the area of a roadnetwork. For example, the submap comparison component 250 can recognizewhen a classified and/or recognized object/feature output from thesensor analysis sub-system 230 is new or different as compared to thefeature set 223 of the same submap. The submap comparison component 250can compare, for example, objects and features of the vehicle's scene,as well as road surface conditions/features and/or lighting conditions,in order to generate a submap feature update 255 for the correspondingsubmap.

The update and versioning component 244 of the submap service manager240 can implement processes to write the updates to the submap database248 for the corresponding submap. Additionally, the update andversioning component 244 can implement versioning for an updated submapso that a given submap is consistent with submaps of adjacent areasbefore such submaps are transmitted to the vehicles. In order to versiona given submap, some examples provide for the update and versioningcomponent 244 to create a copy of the submap to which changes are made,so that two or more versions of a given submap exist in the submapdatabase 248. This allows for different vehicles to carry differentversions of submaps, so that updates to submaps are not required to bedistributed to all vehicles at once, but rather can be rolled outprogressively, according to logic that can optimize bandwidth, networkresources and vehicle availability to receive the updates.

When submaps are updated to carry additional or new data reflecting achange in the area represented by the submap, the change may be discreteto reflect only one, or a specific number of submaps. In somevariations, however, the submap feature update 255 can affect othersubmaps, such as adjacent submaps (e.g., lane detour). Additionally, theupdate and versioning component 244 can receive submap systematicupdates 259 from external sources. The submap systematic updates 259 mayaffect submap by class, requiring replacement of submaps orre-derivation of data layers. For example, the systematic updates 259may require vehicles to implement specific algorithms or protocols inorder to process a data layer of the submap. In some examples, theversioning component 244 can also configure the submaps to carry programfiles, and/or data to enable vehicles to locate and implement programfiles, for purpose of processing the updated submaps.

When systematic updates occur to a group or collection of submaps, theupdate and versioning component 244 can create new submap versions for acollection or group of submaps at a time, so that vehicles can receivesets of new submaps 237 which are updated and versioned to be compatiblewith the vehicle (e.g., when the vehicle is also updated to process thesubmap) and with each other. The update and versioning component 244can, for example, ensure that the new (and updated) submaps 237 can bestitched by the vehicles into routes that the respective vehicles canuse to traverse a road network for a given geographic region, beforesuch updated submaps are communicated to the vehicles.

Submap Data Aggregation

FIG. 3 illustrates a submap data aggregation that stores and linksmultiple versions of submaps, collectively representing linked roadwaysegments for a geographic region, according to one or more examples. InFIG. 3, a submap data aggregation 300 may be implemented as, forexample, the collection of stored submaps 105 on an autonomous vehicle10, as described with an example of FIG. 1. With further reference to anexample of FIG. 3, the submap data aggregation 300 may logicallystructure submaps to include a submap definition 311, 312, as well asone or more data layers 340 which can provide information items such assubmap feature set 113 (see FIG. 1). Among other benefits, the submapdata aggregation 300 enables the submap associated with a given roadsegment to be updated independent of updates to submaps for adjacentroad segments. In one example, individual road segments of a roadnetwork can be represented by multiple versions of a same submap (e.g.,as defined for a particular road segment), with each version includingan update or variation that is not present with other versions of thesame submap.

According to some examples, each submap definition 311, 312 can includean association or grouping of data sets (e.g., files in a folder, tablewith rows and columns, etc.) which collectively identify to a roadsegment. Each submap definition 311, 312 may also correlate to a submapversion and a submap geographic coordinate set, such as to define thegeometric boundary of the submap. The data sets that are associated orgrouped to a particular submap may include a semantic label layer 303, aroad surface layer 304, a perception layer 305 and a localization layer306. The type of data layers which are recited as being included withindividual submaps serve as examples only. Accordingly, variations toexamples described may include more or fewer data layers, as well asdata layers of alternative types.

The submap data aggregation 300 of a road segment may be processed by,for example, the submap processing component 120 contemporaneously withthe vehicle 10 traversing a corresponding portion of the road segment.The various data layers of individual submaps are processed tofacilitate, for example, the AV control system 400 in understanding theroad segment and the surrounding area. According to examples, thelocalization layer 306 includes sensor data, such as imagelets (ascaptured by, for example, stereoscopic cameras of the vehicle 10)arranged in a three-dimensional point cloud, to represent the view of agiven scene at any one of multiple possible positions within the submap.The localization layer 306 may thus include data items that are storedas raw or processed image data, in order to provide a point ofcomparison for the localization component 122 of the submap processingcomponent 110.

With reference to an example of FIG. 1, the localization component 122may use the localization layer 306 to perform localization, in order todetermine a pinpoint or highly granular location, along with a pose ofthe vehicle at a given moment in time. According to some examples, thelocalization layer 306 may provide a three-dimensional point cloud ofimagelets and/or surfels. Depending on the implementation, the imageletsor surfels may represent imagery captured through Lidar, stereoscopiccameras, a combination of two-dimensional cameras and depth sensors, orother three-dimensional sensing technology. In some examples, thelocalization component 122 can determine precise location and pose forthe vehicle by comparing image data, as provided from the current sensorstate 93 of the vehicle, with the three-dimensional point cloud of thelocalization layer 306.

In some examples, the perception layer 305 can include image data,labels or other data sets which mark static or persistent objects. Withreference to an example of FIG. 1, the perception component 124 may usethe perception layer 305 to subtract objects identified through theperception layer 305 from a scene as depicted by the current sensorstate 493 (see FIG. 1). In this way, the submap processing component 120can use the perception layer 305 to detect dynamic objects. Among otheroperations, the vehicle 10 can use the perception layer 305 to detectdynamic objects for purpose of avoiding collisions and/or planningtrajectories within a road segment of the particular submap.

The road surface layer 304 can include, for example, sensor datarepresentations and/or semantic labels that are descriptive of the roadsurface. The road surface layer 304 can identify, for example, thestructure and orientation of the roadway, the lanes of the roadway, andthe presence of obstacles which may have previously been detected on theroadway, predictions of traffic flow patterns, trajectoryrecommendations and/or various other kinds of information.

The label layer 303 can identify semantic labels for the roadway andarea surrounding the road segment of the submap. This may include labelsthat identify actions needed for following signage or traffic flow.

The individual submaps 311, 312 may also include organization data 302,which can identify a hierarchy or dependency as between individual datalayers of the submap. For example, in some examples, the localizationlayer 306 and the perception layer 305 may be dependent on the roadsurface layer 304, as the data provided by the respective localizationand perception layers 306, 305 would be dependent on, for example, acondition of the road surface.

In an example of FIG. 3, the submap versions for a common road segmentare distinguished through lettering (312A-312C). Each submap version maybe distinguished from other versions by, for example, the processinglogic to be used with the submap, the submap data structure (e.g., theinterdependencies of the data layers), the format of the data layers,and/or the contents of the respective data layers. In some examples,each submap 311, 312 may utilize models, algorithms, or other logic(shown as model 315) in connection with processing data for each of thedata layers. The logic utilized to process data layers within a submapmay differ. Additionally, different logic may be utilized for processingdata layers of the submap for different purposes. Accordingly, the datalayers of the individual submaps may be formatted and/or structured soas to be optimized for specific logic.

According to some examples, the structure of the submap data aggregation300 permits individual submaps to be updated independent of othersubmaps (e.g., submaps of adjacent road segments). For example,individual submaps for a geographic region can be updated selectivelybased on factors such as the occurrence of events which affect onesubmap over another. When submaps are updated, the submap in itsentirety may be replaced by an updated submap. As an addition orvariation, components of the submap (e.g., a data layer) can be modifiedor replaced independent of other components of the submap. The updatingof the submap can change, for example, information conveyed in one ormore data layers (e.g., perception layer 305 reflects a new building,road surface layer 304 identifies road construction, etc.), thestructure or format of the data layers (e.g., such as to accommodate newor updated logic of the vehicle 10 for processing the data layer), theorganizational data (e.g., a submap may alter the perception layer 305to be dependent on the localization layer 306), or the type andavailability of data layers (e.g., more or fewer types of semanticlabels for the label layer 303).

According to some examples, each submap includes an identifier thatincludes a versioning identifier (“versioning ID 325”). When a submapfor a particular road segment is updated, a new version of a submap iscreated, and the identifier of the submap is changed to reflect a newversion. In one implementation, the version identifier can be mapped toa record that identifies the specific component version and/or date ofupdate. In another implementation, the versioning ID 325 is encoded toreflect the mapping of the updated submap received for that version ofthe submap.

The data sets that are associated or grouped to a particular submap mayalso include a connector data set 308 (e.g., edge) that links theparticular version of the submap to a compatible version of the submapfor the adjacent road segment. Each connector data set 308 may linkversions for submaps of adjacent road segments using logic or encodingthat identifies and matches compatible submap updates. In oneimplementation, the connector data sets 308 use the versioning ID 325 ofthe individual submaps to identify compatibility amongst adjacentsubmaps and versions thereof. The logic utilized in forming theconnector data sets 308 can account for the type of nature of theupdate, such as the particular data layer or component that is updatedwith a particular submap version. In some examples, when the update tothe submap affects, for example, an algorithm or model for determiningthe interdependent data sets, the versioning ID 325 can reflectcompatibility with only those submaps that utilize the same algorithm ormodel 315. When the update to the submap affects the structure of a datalayer such as localization layer 306 or perception layer 305, theversioning of the data layer may reflect, for example, that the specificsubmap version is compatible with multiple other submap versions whichprovide for the same data layer structures.

With reference to an example of FIG. 1, the connector data sets 308 maycause the submap retrieval and processing components 110, 120 toretrieve and/or process a particular submap version, based oncompatibility to the current submaps 125 that are processed. In thisway, a vehicle that utilizes a particular submap version can ensure thatthe submap of the vehicle's next road segment is of a compatibleversion. Among other benefits, the use of connector data sets 308 enableupdates (e.g., such as from the submap network service 200) to begenerated for numerous vehicles, and then distributed on a roll-outbasis, based on opportunity and availability of individual vehicles toreceive updates. The roll out of the submaps can be performed so thatvehicles, which may receive submap updates early or late in the process,can have a series of compatible submaps for use when traversing the roadnetwork of a given region.

System Description

FIG. 4 illustrates an example of a control system for an autonomousvehicle. In an example of FIG. 4, a control system 400 is used toautonomously operate a vehicle 10 in a given geographic region for avariety of purposes, including transport services (e.g., transport ofhumans, delivery services, etc.). In examples described, an autonomouslydriven vehicle can operate without human control. For example, in thecontext of automobiles, an autonomously driven vehicle can steer,accelerate, shift, brake and operate lighting components. Somevariations also recognize that an autonomous-capable vehicle can beoperated either autonomously or manually.

In one implementation, the AV control system 400 can utilize specificsensor resources in order to intelligently operate the vehicle 10 inmost common driving situations. For example, the AV control system 400can operate the vehicle 10 by autonomously steering, accelerating andbraking the vehicle 10 as the vehicle progresses to a destination. Thecontrol system 400 can perform vehicle control actions (e.g., braking,steering, accelerating) and route planning using sensor information, aswell as other inputs (e.g., transmissions from remote or local humanoperators, network communication from other vehicles, etc.).

In an example of FIG. 4, the AV control system 400 includes a computeror processing system which operates to process sensor data that isobtained on the vehicle with respect to a road segment that the vehicleis about to drive on. The sensor data can be used to determine actionswhich are to be performed by the vehicle 10 in order for the vehicle tocontinue on a route to a destination. In some variations, the AV controlsystem 400 can include other functionality, such as wirelesscommunication capabilities, to send and/or receive wirelesscommunications with one or more remote sources. In controlling thevehicle, the AV control system 400 can issue instructions and data,shown as commands 85, which programmatically controls variouselectromechanical interfaces of the vehicle 10. The commands 85 canserve to control operational aspects of the vehicle 10, includingpropulsion, braking, steering, and auxiliary behavior (e.g., turninglights on).

Examples recognize that urban driving environments present significantchallenges to autonomous vehicles. In particular, the behavior ofobjects such as pedestrians, bicycles, and other vehicles can vary basedon geographic region (e.g., country or city) and locality (e.g.,location within a city). Additionally, examples recognize that thebehavior of such objects can vary based on various other events, such astime of day, weather, local events (e.g., public event or gathering),season, and proximity of nearby features (e.g., crosswalk, building,traffic signal). Moreover, the manner in which other drivers respond topedestrians, bicyclists and other vehicles varies by geographic regionand locality.

Accordingly, examples provided herein recognize that the effectivenessof autonomous vehicles in urban settings can be limited by thelimitations of autonomous vehicles in recognizing and understanding howto process or handle the numerous daily events of a congestedenvironment.

The autonomous vehicle 10 can be equipped with multiple types of sensors401, 403, 405, which combine to provide a computerized perception of thespace and environment surrounding the vehicle 10. Likewise, the AVcontrol system 400 can operate within the autonomous vehicle 10 toreceive sensor data from the collection of sensors 401, 403, 405, and tocontrol various electromechanical interfaces for operating the vehicleon roadways.

In more detail, the sensors 401, 403, 405 operate to collectively obtaina complete sensor view of the vehicle 10, and further to obtaininformation about what is near the vehicle, as well as what is near orin front of a path of travel for the vehicle. By way of example, thesensors 401, 403, 405 include multiple sets of cameras sensors 401(video camera, stereoscopic pairs of cameras or depth perceptioncameras, long range cameras), remote detection sensors 403 such asprovided by radar or Lidar, proximity or touch sensors 405, and/or sonarsensors (not shown).

Each of the sensors 401, 403, 405 can communicate with, or utilize acorresponding sensor interface 410, 412, 414. Each of the sensorinterfaces 410, 412, 414 can include, for example, hardware and/or otherlogical component which is coupled or otherwise provided with therespective sensor. For example, the sensors 401, 403, 405 can include avideo camera and/or stereoscopic camera set which continually generatesimage data of an environment of the vehicle 10. As an addition oralternative, the sensor interfaces 410, 412, 414 can include a dedicatedprocessing resource, such as provided with a field programmable gatearray (“FPGA”) which receives and/or processes raw image data from thecamera sensor.

In some examples, the sensor interfaces 410, 412, 414 can include logic,such as provided with hardware and/or programming, to process sensordata 99 from a respective sensor 401, 403, 405. The processed sensordata 99 can be outputted as sensor data 411. As an addition orvariation, the AV control system 400 can also include logic forprocessing raw or pre-processed sensor data 99.

According to one implementation, the vehicle interface subsystem 90 caninclude or control multiple interfaces to control mechanisms of thevehicle 10. The vehicle interface subsystem 90 can include a propulsioninterface 92 to electrically (or through programming) control apropulsion component (e.g., a gas pedal), a steering interface 94 for asteering mechanism, a braking interface 96 for a braking component, andlighting/auxiliary interface 98 for exterior lights of the vehicle. Thevehicle interface subsystem 90 and/or control system 400 can include oneor more controllers 84 which receive one or more commands 85 from the AVcontrol system 400. The commands 85 can include route information 87 andone or more operational parameters 89 which specify an operational stateof the vehicle (e.g., desired speed and pose, acceleration, etc.).

The controller(s) 84 generate control signals 419 in response toreceiving the commands 85 for one or more of the vehicle interfaces 92,94, 96, 98. The controllers 84 use the commands 85 as input to controlpropulsion, steering, braking and/or other vehicle behavior while theautonomous vehicle 10 follows a route. Thus, while the vehicle 10 mayfollow a route, the controller(s) 84 can continuously adjust and alterthe movement of the vehicle in response receiving a corresponding set ofcommands 85 from the AV control system 400. Absent events or conditionswhich affect the confidence of the vehicle in safely progressing on theroute, the AV control system 400 can generate additional commands 85from which the controller(s) 84 can generate various vehicle controlsignals 419 for the different interfaces of the vehicle interfacesubsystem 90.

According to examples, the commands 85 can specify actions that are tobe performed by the vehicle 10. The actions can correlate to one ormultiple vehicle control mechanisms (e.g., steering mechanism, brakes,etc.). The commands 85 can specify the actions, along with attributessuch as magnitude, duration, directionality or other operationalcharacteristic of the vehicle 10. By way of example, the commands 85generated from the AV control system 400 can specify a relative locationof a road segment which the autonomous vehicle 10 is to occupy while inmotion (e.g., change lanes, move to center divider or towards shoulder,turn vehicle etc.). As other examples, the commands 85 can specify aspeed, a change in acceleration (or deceleration) from braking oraccelerating, a turning action, or a state change of exterior lightingor other components. The controllers 84 translate the commands 85 intocontrol signals 419 for a corresponding interface of the vehicleinterface subsystem 90. The control signals 419 can take the form ofelectrical signals which correlate to the specified vehicle action byvirtue of electrical characteristics that have attributes for magnitude,duration, frequency or pulse, or other electrical characteristics.

In an example of FIG. 4, the AV control system 400 includes SIPS 100,including localization component 122 and perception component 124. TheAV control system 400 may also include route planner 422, motionplanning component 424, event logic 474, prediction engine 426, and avehicle control interface 428. The vehicle control interface 428represents logic that communicates with the vehicle interface subsystem90, in order to issue commands 85 that control the vehicle with respectto, for example, steering, lateral and forward/backward acceleration andother parameters. The vehicle control interface 428 may issue thecommands 85 in response to determinations of various logical componentsof the AV control system 400.

In an example of FIG. 4, the SIPS 100 is shown as a sub-component of theAV control system 400. In variations, the components and functionalityof the SIPS 100 can be distributed with other components in the vehicle.The SIPS 100 can utilize a current sensor state 93 of the vehicle 10, asprovided by sensor data 411. The current sensor state 93 can include rawor processed sensor data obtained from Lidar, stereoscopic imagery,and/or depth sensors. As described with an example of FIG. 1, the SIPS100 may provide localization output 121 (including localizationcoordinate 117 and pose 119, as shown with an example of FIG. 1) to oneor more components of the AV control system 400. The localization output121 can correspond to, for example, a position of the vehicle within aroad segment. The localization output 121 can be specific in terms ofidentifying, for example, any one or more of a driving lane that thevehicle 10 is using, the vehicle's distance from an edge of the road,the vehicle's distance from the edge of the driving lane, and/or adistance of travel from a point of reference for the particular submap.In some examples, the localization output 121 can determine the relativelocation of the vehicle 10 within a road segment, as represented by asubmap, to within less than a foot, or to less than a half foot.

Additionally, the SIPS 100 may signal perception output 129 to one ormore components of the AV control system 400. The perception output 129may utilize, for example, the perception layer 305 (see FIG. 3) tosubtract objects which are deemed to be persistent from the currentsensor state 93 of the vehicle. Objects which are identified through theperception component 124 can be perceived as being static or dynamic,with static objects referring to objects which are persistent orpermanent in the particular geographic region. The perception component124 may, for example, generate perception output 129 that is based onsensor data 411 which exclude predetermined static objects. Theperception output 129 can correspond to interpreted sensor data, such as(i) image, sonar or other electronic sensory-based renderings of theenvironment, (ii) detection and classification of dynamic objects in theenvironment, and/or (iii) state information associated with individualobjects (e.g., whether object is moving, pose of object, direction ofobject). The perception component 124 can interpret the sensor data 411for a given sensor horizon. In some examples the perception component124 can be centralized, such as residing with a processor or combinationof processors in a central portion of the vehicle. In other examples,the perception component 124 can be distributed, such as onto the one ormore of the sensor interfaces 410, 412, 414, such that the outputtedsensor data 411 can include perceptions.

The motion planning component 424 can process input, which includes thelocalization output 121 and the perception output 129, in order todetermine a response trajectory 425 which the vehicle may take to avoida potential hazard. The motion planning component 424 includes logic todetermine one or more trajectories, or potential trajectories of movingobjects in the environment of the vehicle. When dynamic objects aredetected, the motion planning component 424 determines a responsetrajectory 425, which can be directed to avoiding a collision with amoving object. In some examples, the response trajectory 425 can specifyan adjustment to the vehicle's speed (e.g., vehicle in front slowingdown) or to the vehicle's path (e.g., swerve or change lanes in responseto bicyclist). The response trajectory 425 can be received by thevehicle control interface 428 in advancing the vehicle forward. In someexamples, the motion planning component 424 associates a confidencevalue with the response trajectory 425, and the vehicle controlinterface 428 may implement the response trajectory 425 based on theassociated confidence value. As described below, the motion planningcomponent 424 may also characterize a potential event (e.g., by type,severity), and/or determine the likelihood that a collision or otherevent may occur unless a response trajectory 425 is implemented.

In some examples, the motion planning component 424 may include aprediction engine 426 to determine one or more types of predictions,which the motion planning component 424 can utilize in determining theresponse trajectory 425. In some examples, the prediction engine 426 maydetermine a likelihood that a detected dynamic object will collide withthe vehicle, absent the vehicle implementing a response trajectory toavoid the collision. As another example, the prediction engine 426 canidentify potential points of interference or collision by unseen oroccluded objects on a portion of the road segment in front of thevehicle. The prediction engine 426 may also be used to determine alikelihood as to whether a detected dynamic object can collide orinterfere with the vehicle 10.

In some examples, the motion planning component 424 includes event logic474 to detect conditions or events, such as may be caused by weather,conditions or objects other than moving objects (e.g., potholes, debris,road surface hazard, traffic, etc.). The event logic 474 can use thevehicle's sensor state 93, localization output 121, perception output129 and/or third-party information to detect such conditions and events.Thus, the event logic 474 detects events which, if perceived correctly,may in fact require some form of evasive action or planning. In someexamples, response action 425 may include input for the vehicle todetermine a new vehicle trajectory 479, or to adjust an existing vehicletrajectory 479, either to avoid or mitigate a potential hazard. By wayof example, the vehicle response trajectory 425 can cause the vehiclecontrol interface 428 to implement a slight or sharp vehicle avoidancemaneuver, using a steering control mechanism and/or braking component.

The route planner 422 can determine a route 421 for a vehicle to use ona trip. In determining the route 421, the route planner 422 can utilizea map database, such as provided over a network through a map service399. Based on input such as destination and current location (e.g., suchas provided through a satellite navigation component), the route planner422 can select one or more route segments that collectively form a pathof travel for the autonomous vehicle 10 when the vehicle in on a trip.In one implementation, the route planner 422 can determine route input473 (e.g., route segments) for a planned route 421, which in turn can becommunicated to the vehicle control 428.

In an example of FIG. 4, the vehicle control interface 428 includescomponents to operate the vehicle on a selected route 421, and tomaneuver the vehicle based on events which occur in the vehicle'srelevant surroundings. The vehicle control interface 428 can include aroute following component 467 to receive a route input 473 thatcorresponds to the selected route 421. Based at least in part on theroute input 473, the route following component 467 can determine a routetrajectory component 475 that corresponds to a segment of the selectedroute 421. A trajectory following component 469 can determine or selectthe vehicle's trajectory 479 based on the route trajectory 475 and inputfrom the motion planning component 424 (e.g., response trajectory 425when an event is detected). In a scenario where the vehicle is drivingautonomously without other vehicles or objects, the route trajectorycomponent 475 may form a sole or primary basis of the vehicle trajectory479. When dynamic objects or events are detected for avoidance planningby the motion planning component 424, the trajectory following component469 can select or determine an alternative trajectory based on theresponse trajectory 425. For example, the response trajectory 425 canprovide an alternative to the route trajectory 475 for a short durationof time, until an event is avoided. The selection and/or use of theresponse trajectory 425 (or response trajectories) can be based on theconfidence, severity and/or type of object or event detected by themotion planning component 424. Additionally, the selection and/or use ofthe response trajectory 425 can be weighted based on the confidencevalue associated with the determinations, as well as the severity, type,and/or likelihood of occurrence. The vehicle control interface 428 caninclude a command interface 488, which uses the vehicle trajectory 479to generate the commands 85 as output to control components of thevehicle 10. The commands can further implement driving rules and actionsbased on various context and inputs.

Hardware Diagrams

FIG. 5 is a block diagram of a vehicle system on which an autonomousvehicle control system may be implemented. According to some examples, avehicle system 500 can be implemented using a set of processors 504,memory resources 506, multiple sensors interfaces 522, 528 (orinterfaces for sensors) and location-aware hardware such as shown bysatellite navigation component 524. In an example shown, the vehiclesystem 500 can be distributed spatially into various regions of avehicle. For example, a processor bank 504 with accompanying memoryresources 506 can be provided in a vehicle trunk. The various processingresources of the vehicle system 500 can also include distributed sensorprocessing components 534, which can be implemented usingmicroprocessors or integrated circuits. In some examples, thedistributed sensor logic 534 can be implemented using field-programmablegate arrays (FPGA).

In an example of FIG. 5, the vehicle system 500 further includesmultiple communication interfaces, including a real-time communicationinterface 518 and an asynchronous communication interface 538. Thevarious communication interfaces 518, 538 can send and receivecommunications to other vehicles, central services, human assistanceoperators, or other remote entities for a variety of purposes. In thecontext of FIG. 1 and FIG. 4, for example, the SIPS 100 and the AVcontrol system 400 can be implemented using vehicle system 500, as withan example of FIG. 5. In one implementation, the real-time communicationinterface 518 can be optimized to communicate information instantly, inreal-time to remote entities (e.g., human assistance operators).Accordingly, the real-time communication interface 518 can includehardware to enable multiple communication links, as well as logic toenable priority selection.

The vehicle system 500 can also include a local communication interface526 (or series of local links) to vehicle interfaces and other resourcesof the vehicle 10. In one implementation, the local communicationinterface 526 provides a data bus or other local link toelectro-mechanical interfaces of the vehicle, such as used to operatesteering, acceleration and braking, as well as to data resources of thevehicle (e.g., vehicle processor, OBD memory, etc.). The localcommunication interface 526 may be used to signal commands 535 to theelectro-mechanical interfaces in order to control operation of thevehicle.

The memory resources 506 can include, for example, main memory, aread-only memory (ROM), storage device, and cache resources. The mainmemory of memory resources 506 can include random access memory (RAM) orother dynamic storage device, for storing information and instructionswhich are executable by the processors 504.

The processors 504 can execute instructions for processing informationstored with the main memory of the memory resources 506. The main memorycan also store temporary variables or other intermediate informationwhich can be used during execution of instructions by one or more of theprocessors 504. The memory resources 506 can also include ROM or otherstatic storage device for storing static information and instructionsfor one or more of the processors 504. The memory resources 506 can alsoinclude other forms of memory devices and components, such as a magneticdisk or optical disk, for purpose of storing information andinstructions for use by one or more of the processors 504.

One or more of the communication interfaces 518 can enable theautonomous vehicle to communicate with one or more networks (e.g.,cellular network) through use of a network link 519, which can bewireless or wired. The vehicle system 500 can establish and use multiplenetwork links 519 at the same time. Using the network link 519, thevehicle system 500 can communicate with one or more remote entities,such as network services or human operators. According to some examples,the vehicle system 500 stores submaps 505, as well as submap controlsystem instructions 507 for implementing the SIPS 100 (see FIG. 1). Thevehicle system 500 may also store AV control system instructions 509 forimplementing the AV control system 400 (see FIG. 4). During runtime(e.g., when the vehicle is operational), one or more of the processors504 execute the submap processing instructions 507, including theprediction engine instructions 515, in order to implement functionalitysuch as described with an example of FIG. 1.

In operating the autonomous vehicle 10, the one or more processors 504can execute AC control system instructions 509 to operate the vehicle.Among other control operations, the one or more processors 504 mayaccess data from a road network 525 in order to determine a route,immediate path forward, and information about a road segment that is tobe traversed by the vehicle. The road network can be stored in thememory 506 of the vehicle and/or received responsively from an externalsource using one of the communication interfaces 518, 538. For example,the memory 506 can store a database of roadway information for futureuse, and the asynchronous communication interface 538 can repeatedlyreceive data to update the database (e.g., after another vehicle does arun through a road segment).

FIG. 6 is a block diagram of a network service or computer system onwhich some embodiments may be implemented. According to some examples, acomputer system 600, such as shown with an example of FIG. 2, may beused to implement a submap service or other remote computer system, suchas shown with an example of FIG. 2.

In one implementation, the computer system 600 includes processingresources, such as one or more processors 610, a main memory 620, aread-only memory (ROM) 630, a storage device 640, and a communicationinterface 650. The computer system 600 includes at least one processor610 for processing information and the main memory 620, such as a randomaccess memory (RAM) or other dynamic storage device, for storinginformation and instructions to be executed by the processor 610. Themain memory 620 also may be used for storing temporary variables orother intermediate information during execution of instructions to beexecuted by the processor 610. The computer system 600 may also includethe ROM 630 or other static storage device for storing staticinformation and instructions for the processor 610. The storage device640, such as a magnetic disk or optical disk, is provided for storinginformation and instructions. For example, the storage device 640 cancorrespond to a computer-readable medium that stores instructions formaintaining and distributing submaps to vehicles, such as described withexamples of FIG. 2 and FIG. 8. In such examples, the computer system 600can store a database of submaps 605 for a geographic region, with eachsubmap being structured in accordance with one or more examplesdescribed below. The memory 620 may also store instructions for managingand distributing submaps (“submap instructions 615”). For a givengeographic region, individual submaps 605 may represent road segmentsand their surrounding area. The processor 604 may execute the submapinstructions 615 in order to perform any of the methods such asdescribed with FIG. 8.

The communication interface 650 can enable the computer system 600 tocommunicate with one or more networks 680 (e.g., cellular network)through use of the network link (wirelessly or using a wire). Using thenetwork link, the computer system 600 can communicate with a pluralityof user-vehicles, using, for example, wireless network interfaces whichmay be resident on the individual vehicles.

The computer system 600 can also include a display device 660, such as acathode ray tube (CRT), an LCD monitor, or a television set, forexample, for displaying graphics and information to a user. An inputmechanism 670, such as a keyboard that includes alphanumeric keys andother keys, can be coupled to the computer system 600 for communicatinginformation and command selections to the processor 610. Othernon-limiting, illustrative examples of the input mechanisms 670 includea mouse, a trackball, touch-sensitive screen, or cursor direction keysfor communicating direction information and command selections to theprocessor 610 and for controlling cursor movement on the display 660.

Some of the examples described herein are related to the use of thecomputer system 600 for implementing the techniques described herein.According to one example, those techniques are performed by the computersystem 600 in response to the processor 610 executing one or moresequences of one or more instructions contained in the main memory 620.Such instructions may be read into the main memory 620 from anothermachine-readable medium, such as the storage device 640. Execution ofthe sequences of instructions contained in the main memory 620 causesthe processor 610 to perform the process steps described herein. Inalternative implementations, hard-wired circuitry may be used in placeof or in combination with software instructions to implement examplesdescribed herein. Thus, the examples described are not limited to anyspecific combination of hardware circuitry and software.

Methodology

FIG. 7 illustrates an example method for operating a vehicle using asubmap system, according to one or more examples. According to examples,the method such as described with FIG. 7 may be implemented usingcomponents such as described with FIGS. 1, 4 and 5. Accordingly, indescribing an example of FIG. 7, reference may be made to elements ofprior examples in order to illustrate a suitable component forperforming a step or sub-step being described.

In one implementation, the autonomous vehicle retrieves a series ofsubmaps (or one or more submaps) from a collection of submaps that arestored in memory (710). The series of submaps may be retrieved for usein controlling the vehicle 10 on a trip. As described with otherexamples, each submap may represent an area of a road network on whichthe vehicle is expected to travel. According to some examples, theindividual submaps of the collection can each include (i) an identifierfrom the collection, (ii) multiple data layers, with each data layerrepresenting a feature set of the area of the road network of thatsubmap, and (iii) a connector data set to link the submap with anothersubmap that represents an adjacent area to the area of the road networkof that submap. By way of example, the retrieval operation may beperformed in connection with the vehicle initiating a trip. In such anexample, the retrieval operation is performed to obtain submaps for thevehicle 10 prior to the vehicle progressing on the trip to the pointwhere a submap is needed. In variations, the vehicle 10 may retrievesubmaps in anticipation of use at a future interval.

In some examples, the retrieval operation is local (712). For example,the submap retrieval process 110 may retrieve submaps from a collectionof submaps 105 that are stored with memory resources of the vehicle. Invariations, the submap retrieval process 110 may retrieve submaps from aremote source (714), such as the submap network service 200, or anothervehicle. For example, the vehicle 10 may communicate wirelessly (e.g.,using a cellular channel) with the submap network service 200, or withanother vehicle 10 which may have submaps to share.

The vehicle 10 can be controlled in its operations during the trip usingthe retrieved submaps (720). For example, the submap processingcomponent 120 of the SIPS 100 can extract data from the various datalayers of each submap, for use as input to the AV control system 400.The AV control system 400 can, for example, utilize the submap tonavigate, plan trajectories, determine and classify dynamic objects,determine response actions, and perform other operations involved indriving across a segment of a road network that corresponds to a submap.

FIG. 8 illustrates an example method for distributing mappinginformation to vehicles of a geographic region for use in autonomousdriving, according to one or more examples. A method such as describedwith FIG. 8 may be implemented using components such as described withFIG. 2 and FIG. 6. Accordingly, in describing an example of FIG. 7,reference may be made to elements of prior examples in order toillustrate a suitable component for performing a step or sub-step beingdescribed.

With reference to an example of FIG. 8, the submap network service 200may maintain a series of submaps which are part of a submap database(810). In one implementation, the submap network service 200 may utilizeservers and network resources to maintain a library of submaps for ageographic region.

In some variations, the submap network service 200 can update submapsindividually and independent of other submaps (820). When such updatesoccur, the submap network service 200 can distribute updated submaps toa population of vehicles in the pertinent geographic region. Eachvehicle may then receive or store an updated set of submaps. The abilityto update and distribute submaps individually, apart from a larger mapof the geographic region, facilitates the submap network service 200 inefficiently and rapidly managing the submap library, and the collectionsof submaps which can be repeatedly communicated to user-vehicles of thepertinent geographic region.

As described with examples of FIG. 1 and FIG. 3, submaps may beversioned to facilitate partial distribution to vehicles of a population(822). Versioning submaps facilitate the submap network service 200 inprogressively implementing global updates to the submaps of a geographicregion. Additionally, versioning submaps enables user-vehicles tooperate utilizing submaps that differ by content, structure, data types,or processing algorithm.

As an addition or alternative, the submap network service 200 maydistribute a series of submaps to multiple vehicles of the geographicregion (830). According to some examples, the distribution of submapsmay be done progressively, to vehicles individually or in small sets,rather than to all vehicles that are to receive the submaps at one time.The versioning of submaps may also facilitate the distribution of newsubmaps, by for example, ensuring that vehicles which receive newsubmaps early or late in the update process can continue to operateusing compatible submap versions.

FIG. 9 illustrates an example method for providing guidance toautonomous vehicles. A method such as described with an example of FIG.9 may be implemented by, for example, a network computer system, such asdescribed with submap network service 200 (see FIG. 2), in connectionwith information provided by autonomous vehicles, such as described withFIG. 1 and FIG. 4. Accordingly, reference may be made to elements ofother examples for purpose of illustrating a suitable component orelement for performing a step or sub-step being described.

According to an example, sensor information is obtained from multipleinstances of at least one autonomous vehicle driving through or past aroad segment which undergoes an event or condition that affects trafficor driving behavior (910). The autonomous vehicle 10 may, for example,be in traffic to encounter the causal condition or event. Alternatively,the autonomous vehicle 10 may capture sensor data of other vehicles thatare encountering the condition or event (e.g., the autonomous vehiclemay travel in an alternative direction). The sensor information maycorrespond to image data (e.g., two-dimensional image data,three-dimensional image data, Lidar, radar, sonar, etc.). The sensorinformation may be received and processed by, for example, the submapnetwork service 200.

The submap network service may use the sensor information to identify adeviation from a normal or permitted driving behavior amongst aplurality of vehicles that utilize the road segment (920). The deviationmay be identified as an aggregation of incidents, where, for example,the driving behavior of vehicles in the population deviate form a normalor permitted behavior. The past incidents can be analyzed through, forexample, statistical analysis (e.g., development of histograms), so thatfuture occurrences of the deviations may be predicted. By way ofexample, a deviation may correspond to an ad-hoc formation of a turnlane. In some cases, the deviation may be a technical violation of lawor driving rules for the geographic region. For example, the deviationmay correspond to a turn restriction that is otherwise permissible, butnot feasible to perform given driving behavior of other vehicles. Asanother example, the deviation may correspond to a reduction ins speedas a result of traffic formation, such as by other vehicles anticipatingtraffic. As another example, the deviation may include the formation ofa vehicle stopping space that other vehicles utilize, but which isotherwise impermissible. Alternatively, the deviation may includeelimination of a vehicle parking space that is permissible, but notfeasible to access given driving behavior of other vehicles.

In order to identify the deviation, the submap data analysis 230 mayextract vehicle sensor data 243 transmitted form a vehicle, then plotthe localization position of the vehicle to determine when and where theautonomous vehicle 10 occupied a lane that crossed a midline of theroad, or a shoulder on the side of the road. As an alternative orvariation, the submap data analysis 230 may perform image (or othersensor data) analysis to identify, for example, vehicles standing stillor conglomerating in places of the road network to block access forturning or parking spots.

In some examples, the submap network service 200 determines thedeviation as being a pattern behavior (922). The pattern behavior may betemporal, such as reoccurring on specific days of week or time.

In variations, the behavior may be responsive to certain events orconditions (924). For example, snowfall or rain may be observed to causevehicles on a road segment to drive towards a center divider.

The instructions set for one or more autonomous vehicles may be updatedto enable or facilitate the autonomous vehicles to implement thedeviation (930). In some examples, the instructions may be updated byinclusion of parameters, sensor data or other information in the submapthat encompasses the location of the deviation. As an addition orvariation, the instructions may be updated by relaxing driving rules forthe autonomous vehicle 10 to permit driving behavior which wouldotherwise be considered impermissible or constrained.

In some examples, the submap may include parameters or instructions toindicate when the deviation is anticipated, or when alternative drivingbehavior to account for the deviation is permissible (932). For example,the deviation may be patterned in time, and the submap for the vehiclemay weight against the vehicle implementing the deviation unless withintime slots when the driving behavior of other vehicles warrants thedeviation.

FIG. 10 illustrates an example sensor processing sub-system for anautonomous vehicle, according to one or more embodiments. According tosome examples, a sensor processing sub-system 1000 may be implemented aspart of an AV control system 400 (see FIG. 4). In some examples, thesensor processing subsystem 1000 can be implemented as the submapinformation processing system 100 (e.g., see FIGS. 1 and 4). Invariations, the sensor processing subsystem 1000 may be implementedindependent of submaps.

According to an example of FIG. 10, the sensor processing subsystem 1000includes a localization component 1024, a perception component 1022, andimage processing logic 1038. The localization component 1024 and/or theperception component 1022 may each utilize the image processing logic1038 in order to determine a respective output. In particular, thelocalization component 1024 and the perception component 1022 may eachcompare current sensor state data 493, including current image data 1043captured by onboard cameras of the vehicle, with prior sensor state data1029. The current image data 1043 may include, for example, passiveimage data, such as provided with depth images generated from pairs ofstereoscopic cameras and two dimensional images (e.g., long rangecameras). The current image data 1043 may also include active imagedata, such as generated by Lidar, sonar, or radar.

In some examples such as described with FIG. 1 through FIG. 3, priorsensor state 1029 may be provided through use of submaps, which maycarry or otherwise provide data layers corresponding to specific typesof known sensor data sets (or features) for a given area of a roadsegment. In variations, prior sensor state 1029 may be stored and/oraccessed in another form or data stricture, such as in connection withlatitude and longitude coordinates provided by a satellite navigationcomponent.

The localization component 1024 may determine a localization output 1021based on comparison of the current sensor state 493 and prior sensorstate 1029. The localization output 1021 may include a locationcoordinate 1017 and a pose 1019. In some examples, the locationcoordinate 1017 may be with respect to a particular submap which thevehicle 10 is deemed to be located in (e.g., such as when the vehicle 10is on trip). The pose 1019 may also be with respect to a predefinedorientation, such as the direction of the road segment.

According to some examples, the localization component 1024 determinesthe localization output 1021 using the prior sensor state 1029 of anarea of the vehicle. The prior sensor state 1029 may be distributed aselements that reflect a sensor field of view about a specific locationwhere the sensor data was previously obtained. When distributed aboutthe sensor field of view, the sensor information provided by the priorsensor state 1029 can be said to be in the form of a point cloud 1035.In some examples, the point cloud 1035 of prior sensor information maybe substantially two-dimensional, spanning radially (e.g., 180 degrees,360 degrees about, for example, a reference location that is in front ofthe vehicle). In variations, the point cloud 1035 of prior sensorinformation may be three-dimensional, occupying a space in front and/oralong the sides of the vehicle. Still further, in other variations, thepoint cloud 1035 of prior sensor information may extend in two or threedimensions behind the vehicle. For example, the prior sensor state 1029may be provided as part of a submap that includes a layer of imageletsarranged in a point cloud. The imagelets may include, for example,passive image data sets, or image sets collected from a Lidar componentof the vehicle. The individual imagelets of the prior sensor state 1029may each be associated with a precise coordinate or position,corresponding to a location of sensor devices that captured sensorinformation of the prior state information. In some variations, theimagelets of prior sensor state 1029 may also reference a pose,reflecting an orientation of the sensor device that captured the priordata. In some variations, the imagelets may also be associated withlabels, such as semantic labels which identify a type or nature of anobject depicted in the imagelet, or a classification (e.g., imageletdepicts a static object). Still further, the imagelets may be associatedwith a priority or weight, reflecting, for example, a reliability oreffectiveness of the imagelet for purpose of determining thelocalization output 1021.

In some variations, the prior sensor state 1029 may include multiplepoint clouds 135 for different known and successive locations of a roadsegment, such as provided by a submaps. For example, the prior sensorstate 1029 may include a point cloud 1035 of prior sensor informationfor successive locations of capture, along a roadway segment, where thesuccessive locations are an incremental distance (e.g., 1 meter) apart.

As an addition or alternative, the prior sensor state 1029 may includemultiple different point clouds 1035 that reference a common location ofcapture, with variations amongst the point clouds 1035 accounting fordifferent lighting conditions (e.g., lighting conditions such asprovided by weather, time of day, season). In such examples, thelocalization component 1024 may include point cloud selection logic 1032to select the point cloud 1035 of prior sensor information to reflect abest match with a current lighting condition, so that a subsequentcomparison between the prior sensor state 1029 and the current sensorstate 493 does not result in inaccuracies resulting from differences inlighting condition. For example, with passive image data, the variationsamongst the point clouds 1035 of prior sensor information may accountfor lighting variations resulting from time of day, weather, or season.

In some examples, the point cloud selection logic 1032 may select theappropriate point cloud 1035 of prior sensor state information based onan approximate location of the vehicle 10. For example, when the vehicle10 initiates a trip, the point cloud selection logic 1032 may select apoint cloud 1035 of prior sensor information based on a last knownlocation of the vehicle. Alternatively, the point cloud selection logic1032 may select the point cloud 1035 of prior sensor information basedon an approximate location as determined by a satellite navigationcomponent, or through tracking of velocity and time (and optionallyacceleration). As an addition or alternative, the point cloud selectionlogic 1032 may select the appropriate point cloud based on contextualinformation, which identifies or correlates to lighting conditions.Depending on implementation, the selected point cloud 1035 may carryless than 5 imagelets, less than 10 imagelets, or tens or hundreds ofimagelets.

The localization component 1024 may compare the current image data 1043of the current sensor state 493 with that of the selected point cloud1035 in order to determine the localization output 1021. In performingthe comparison, the localization component 1024 may generate, orotherwise create, a fan or field of view for the current sensorinformation, reflecting the scene as viewed from the vehicle at aparticular location. The localization component 1024 may utilize theimage processing logic 1038 to perform image analysis to match featuresof the scene with imagelets of the selected point cloud 1035. Thelocalization component 1024 may use the image processing logic 1038 tomatch portions of the current image data 1043 with individual imageletsof the selected point cloud 1035. In some examples, multiple matchingimagelets are determined for purpose of determining the localizationoutput 1021. For example, in some examples, 3 to 5 matching imageletsare identified and then used to determine the localization output 1021.In variations, more than 10 matching imagelets are identified and usedfor determining the localization output 1021.

In some examples, the localization component 1024 may also include pointselection logic 1034 to select imagelets from the selected point cloud1035 of prior sensor information as a basis of comparison with respectto the current image data 1043. The point selection logic 1034 canoperate to reduce and/or optimize the number of points (e.g., imagets)which are used with each selected point cloud 1035 of prior sensorinformation, thereby reducing a number of image comparisons that areperformed by the localization component 1024 when determining thelocalization output 1021. In one implementation, the point selectionlogic 1034 implements point selection rules 1055. The point selectionrules 1055 can be based on contextual information, such as the weather,time of day, or season. The point selection rules 1055 can also bespecific to the type of sensor data. For passive image data, the pointselection rules 1055 can exclude or de-prioritize imagelets which depictnon-vertical surfaces, such as rooflines or horizontal surfaces, sinceprecipitation, snow and debris can affect the appearance of suchsurfaces. Under the same weather conditions, the point selection rules1055 can also prioritize imagelets which depict vertical surfaces, suchas walls, or signs, as such surfaces tend to preclude accumulation ofsnow or debris. Still further, the point selection rules 1055 canexclude imagelets or portions thereof which depict white when weatherconditions include presence of snow.

Likewise, when Lidar is used, the point selection rules 1055 may selectsurfaces that minimize the effects of rain or snow, such as verticalsurfaces. Additionally, the point selection rules 1055 may avoid orunder-weight surfaces which may be deemed reflective.

The localization component 1024 may use image processing logic 1038 tocompare image data 1043 of the current sensor state 493 against theimagelets of the selected point cloud 1035 in order to determine objectsand features which can form the basis of geometric and spatialcomparison. A perceived geometric and/or spatial differential may bedetermined between objects and/or object features of image data 1043 andimagelets of the selected point cloud 1035. The perceived differentialmay reflect a difference in the location of capture, as between sensordevices (e.g., on-board cameras of the vehicle 10) used to capture thecurrent image data 1043 and the imagelets of the point cloud 1035,representing the prior sensor information. Similarly, the perceivedgeometric differential may reflect a difference in a geometric attribute(e.g., height, width, footprint or shape) of an object or feature thatis depicted in the current image data 1043, as compared to the depictionof the object or feature with the corresponding imagelet of the pointcloud 1035.

The localization component 1024 may include geometric/spatialdetermination logic 1036 to convert the perceived geometric and/orspatial differential into a real-world distance measurement, reflectinga separation distance between the location of capture of the currentimage data 1043 and the location of capture of imagelets of the selectedpoint cloud 1035. As an addition or variation, the geometric/spatialdetermination logic 1036 may convert the perceived geometric and/orspatial differential into a real-world pose or orientation differentialas between the image capturing devices of the current image data 1043and sensor devices uses to capture the imagelets of the selected pointcloud 1035. In some examples, the geometric/spatial determination logic1036 manipulates the perceived object or feature of the current imagedata 1043 so that it matches the shape and/or position of the object orfeature as depicted in imagelets of the selected point cloud 1035. Themanipulation may be used to obtain the values by which the perceivedobject or feature, as depicted by the current image data 1043, differsfrom the previously captured image of the object or feature. Forexample, the perceived object or feature which serves as the point ofcomparison with imagelets of the selected point cloud 1035 may be warped(e.g., enlarged), so that the warped image of object or feature depictedby the current image data 1043 can overlay the image of the object orfeature as depicted by the imagelets of the selected point cloud 1035.

The perception component 1022 may determine a perception output 1025using the current sensor state 493 and prior sensor state 1029. Asdescribed with examples, the perception output 1025 may include (i)identification of image data corresponding to static objects detectedfrom current image data, or (ii) identification of image datacorresponding to non-static objects detected from current image data. Insome examples, the perception output 1025 may also include trackinginformation 1013, indicating past and/or predicted movement of anon-static object.

In some examples, the prior sensor state 1029 may include a staticobject feature set 1037, which includes data sets captured previouslywhich are deemed to depict static objects in the area of the roadsegment. The static objects include permanent objects which are notlikely to change location or appearance over a duration of time. By wayof example, the static objects may include objects which are deemed tohave a height, shape, footprint, and/or visibility that is unchangedover a significant amount of time (e.g., months or years). Thus, forexample, the static objects represented by the static object feature set1037 may include buildings and roadway structures (e.g., fences,overpasses, dividers etc.).

According to some examples, the perception component 1022 uses thestatic object feature set 1037 to identify portions of the current imagedata 1043 which reflect the presence of the static objects. In someexamples, the perception component 1022 may utilize the image processinglogic 1038 to implement image recognition or detection of the staticobject feature depicted by the current image data 1043, using theidentified static object feature set 1037 provided by the prior sensorstate 1029. For example, the static object feature set 1037 may specifysemantic information (e.g., object classification, shape) about a staticobject, as well as a relative location of the static object by pixellocation or image area. The perception component 1022 may use the imageprocessing component 1038 to detect and classify objects in relativeregions of the scene being analyzed, in order to determine if asemantically described static object is present in that image datacorresponding to that portion of the scene. In this way, the perceptioncomponent 1022 may then uses the image processing logic 1038 to detectthe static object from the current image data 1043, using the pixellocation or image area identified by prior sensor state 1029, as well asthe object shape and/or classification.

Once the static objects are detected from the current image data 1043,the perception component 1022 may then deduce other objects depicted bythe current image data 1043 as being non-static or dynamic.Additionally, the perception component 1022 may detect a non-staticobject as occluding a known static object when the image analysis 1038determines that the pixel location/image area identified for the staticobject feature set 1037 does not depict an object or feature of thatset. When the perception component 1022 determines static objects fromthe current sensor state 493, the perception component 1022 mayimplement object subtraction 1026 so that the presence of the staticobject is ignored in connection with one or more sensor analysisprocesses which may be performed by the sensor processing subsystem1000. For example, the sensor processing subsystem 1000 may subsequentlyperform event detection to track objects which are non-static ordynamic. When pixel data corresponding to static objects are ignored orremoved, subsequent processes such as event detection and tracking maybe improved in that such processes quickly focus image processing onnon-static objects that are of interest.

In some examples, the perception component 1022 may include trackinglogic 1014 which operates to track non-static objects, once such objectsare identified. For example, non-static objects may be sampled forposition information over time (e.g., duration for less than a second).To optimize processing, the perception component 1022 may ignore staticobjects, and focus only on the non-static object(s) during a samplingperiod. This enables the autonomous vehicle 10 to reduce the amount ofcomputation resources needed to track numerous objects which areencountered routinely when vehicles are operated. Moreover, the vehiclecan optimize response time for when a tracked object is a potentialcollision hazard.

In some examples, the tracking logic 1014 calculated a trajectory of thenon-static object. The calculated trajectory can include predictedportions. In some examples, the trajectory can identify, for example,one or more likely paths of the non-static object. Alternatively, thetracking logic 1014 may calculate a worst-case predictive trajectory fora non-static object. For example, the tracking logic 1014 may calculatea linear path as between a current location of a tracked non-staticobject, and a path of the vehicle, in order to determine a time,orientation or velocity of the object for collision to occur. Thetracking logic 1014 may perform the calculations and resample for theposition of the non-static object to see re-evaluate whether theworst-case scenario may be fulfilled. In the context of AV controlsystem 400, the perception output 1025 (shown in FIG. 4 as perception423

As illustrated by examples of FIG. 10, image analysis 1038 includesoperations which can be performed to determine localization output 1021and/or perception output 1025. The image analysis 1038, when applied toeither localization component 1024 or perception component 1022, mayinclude rules or logic to optimize or otherwise improve accuracy and/orease of analysis. In some examples, the image processing 1038 includeswarping logic 1063, which includes rules or models to alter or skewdimensions of detected objects. In context of perception component 1022,a detected image provided by current image data 1043 may be enlarged orskewed in order to determine whether the object appears to match any ofthe static objects 1037 which the prior sensor state 1029 indicatesshould be present. In variations, the static objects 1037 may beidentified by semantic labels, and the image processing component 1038can first warp a detected object from the current image data 1043, andthen classify the detected object to determine if it matches thesemantic label provided as the static object 1037. In the context oflocalization, the warping logic 1063 can warp detected objects in thecurrent image data 1043 and/or prior sensor state 1029 in order todetermine if a match exists as to specific features or sub-features ofthe detected object.

Additional examples recognize that with respect to passive image sensordata, the image analysis 1038 may be negatively affected by lightingconditions, or environmental conditions which may impact the appearanceof objects. Thus, for example, an outcome of image analysis 1038 mayaffect the accuracy of efficiency of the geometric/spatial determination1036, in that lighting variations may features depicted by the currentimage data 1043 may be more or less likely to match with correspondingfeatures depicted by the point cloud of imagelets, based on lightingfactors.

According to some examples, the sensor processing subsystem 1000 mayinclude time and/or place shift transformations 1065 for use incomparing passive image data. The shift transformations 1065 may beapplied by, for example, the image processing logic 1038, when imageprocessing is performed in context of either localization component 1024or perception component 1022. Each transformation 1065 can represent avisual alteration to at least a portion of the current image data 1043and/or prior image data. In some examples, the transformations can bequantitative variations that are applied globally to image data from aparticular scene, or alternatively, to a portion of image data capturedfrom a scene. The individual transformations can alter the appearance ofpassive image data sets (either current or prior sensor sets) withrespect to attributes such as hue, brightness and/or contrast. The imageprocessing 1038 may apply the transformations selectively, when, forexample, a disparity exists between current and past image sets withrespect to hue, brightness, or contrast. Examples recognize that suchdisparity may be the result of, for example, variation in time of day(e.g., vehicle currently driving on road segment at night when sensorinformation was previously captured during day time hours), or change inseason (e.g., vehicle currently driving on road segment during winterwhile sensor information was previously captured during summer). In thecase of passive image data, the disparity in hue, brightness or contrastcan impact the ability of the image processing component 1038 toaccurately perform recognition, thus, for example, hindering the abilityof the vehicle to perform localization or perception operations.

According to some examples, the image processing component 1038 mayselectively use the shift transformations 1065 to better match thecurrent and past image data sets for purpose of comparison andrecognition. For example, the image processing component 1038 may detectdisparity in lighting condition between the current image data 1043 andthe image data provided by the prior sensor state 1029, independent ofimage recognition and/or analysis processes. When such disparity isdetected, the sensor processing subsystem 1000 may select atransformation, which can be applied similar to a filter, to accuratelyalter the current image data 1043 in a manner that best approximatesvisual attributes of the image data contained with the prior sensorstate 1029.

Examples also recognize that in a given geographic region, some roads orportions of the road network will have less sensor data sets as a resultof being less traveled than other roads which may carry more traffic.Moreover, roads and road segments may provide substantial variation asto lighting parameters, given environmental factors such as presence oftrees, buildings and street lights. To account for such variations, theshift transformations 1065 can transform the current image data 1043based on a categorization scheme, such as categories for tree coverage,building coverage, and poor street lighting. For a given road segment,in some implementations, the transformations 1065 may be selected forsegments of roads based on road type (e.g., heavy trees, buildings,absence of street lights). In variations, the transformations 1065 maybe based on prior sensor state data 1029 of adjacent or nearby roadsegments, captured under environmental/lighting conditions thatsufficiently match a current condition of the vehicle 10 traveling alonga less traveled road segment.

FIG. 11 illustrates an example of a vehicle on which an example of FIG.10 is implemented. A vehicle 10 may operate autonomously, meaning thevehicle can drive on a route and navigate without human control orinput. The vehicle 10 can implement, for example, the AV control system400, in order to autonomously navigate on a road network of a givengeographic region. In an example of FIG. 11, the vehicle 10 is shown totraverse a road segment 1102, using a given driving lane 1103, andfurthermore in presence of dynamic objects such as other vehicles 1105and people.

In an example of FIG. 11, the vehicle 10 implements the sensorprocessing subsystem 1000 as part of the AV control system 400 in orderto determine the localization output 121 and the perception output 129.Thus, the sensor processing subsystem 1000 can be implemented as part ofSIPS 100, and further as part of the AV control system 400 of theautonomous vehicle 10. The vehicle 10 may include sensor devices, suchas a camera set 1112, shown as a rooftop camera mount, to capture imagesof a surrounding scene while the vehicle travels down the road segment1102. The camera set 1112 may include, for example, stereoscopic camerapairs to capture depth images of the road segment. In the example shown,the sensor processing subsystem 1000 may be implemented using processorsthat are located in, for example, a trunk of the vehicle 10.

In an example, the sensor processing subsystem 1000 can select a pointcloud of imagelets 1122 which represent the prior sensor state capturedfor the road segment 1102. The imagelets 1122 may include raw image data(e.g., pixel images), processed image data (e.g., feature vector ofselect objects), semantic labels, markers and image data of a particularregion of the scene about a prior location of capture 1115, where theprior location of capture 1115 has a precisely known location. Thesensor processing subsystem 1000 can take current sensor state data,including image data captured by the camera set 1112, and fan thecurrent image data about two or three dimensions. In this way, thecurrent image data can be structured or otherwise identified as imagesegments 1118, which can be compared to prior state imagelets 1122 ofthe selected point. The comparison can calculate the difference betweenthe current location of capture 1125 for the current image segments 1118the prior location of capture 1115 for the prior imagelets 1122. Fromthe comparison, the vehicle 10 can determine the localization output121, 1021, including a localization coordinate 1017 and pose 1019, eachof which may be made in reference to the prior location of capture 1115.

FIG. 12 illustrates an example method for determining a location of avehicle in motion using vehicle sensor data, according to an embodiment.FIG. 13 illustrates a method for determining a location of a vehicle inmotion using image data captured by the vehicle, according to anembodiment. FIG. 14 illustrates a method for determining a location of avehicle in motion using an image point cloud and image data captured bythe vehicle, according to an embodiment. FIG. 15 illustrates an examplemethod in which the perception output is used by a vehicle to process ascene. Example methods such as described with FIG. 12 through FIG. 15may be implemented using components and systems such as described withother examples. In particular, examples of FIG. 12 through FIG. 15 aredescribed in context of being implemented by the AV control system 400,which may include or implement the sensor processing subsystem 1000.Accordingly, reference may be made to elements described with otherfigures for purpose of illustrating suitable components for performing astep or sub-step being described.

With reference to an example of FIG. 12, a collection of submaps may beaccessed by the AV control system 400 of the vehicle in motion (1210).The collection of submaps may be locally accessed and/or retrieved overa network from a remote source (e.g., network service 200). Each submapof the collection may include or be associated with prior sensor statedata 1029 corresponding to sensor data and/or sensor-baseddeterminations of static features for a given road segment. The featuresmay include static objects that may be visible to sensors of a vehicle(e.g., landmarks, structures in view of a vehicle on the roadway),roadway features, signage, and/or traffic lights and signs. The featuresmay be stored as data sets that are associated or provided with a datastructure of a corresponding submap. Each data set may, for example,include a sensor-based signature or feature vector representation of aportion of a scene, as viewed by a specific type of sensor set (e.g.,stereoscopic camera, Lidar, radar, sonar, etc.). Furthermore, the storedsensor data sets may be associated with a reference location of thesubmap, such as the location of the vehicle when the prior sensor datasets were captured. One or multiple types of sensor data sets may beprovided with the prior sensor state data 1029 of the submap. Forexample, the prior sensor state 1029 of a submap may includetwo-dimensional image data, stereoscopic image pair data, Lidar, depthimage, radar and/or sonar, as captured from a particular location of aroad network.

In some examples, the feature set associated with the collection ofsubmaps are developed over time, using the sensor components (e.g.,camera set 1112) of the same vehicle 1110 in prior passes of the roadsegment. In variations, the vehicle in 1110 is part of a larger numberof vehicles, each of which record sensor data and/or feature sets of thesame sensor type(s). As described with an example of FIG. 2, the submapnetwork service 200 may collect and process the sensor data fromindividual vehicles of a fleet, and then share the submaps with updatedfeature sets with other vehicles of the fleet.

At an initial time, the AV control system 400 may determine which submapin a collection of submaps is for the particular road segment on whichthe vehicle 1110 is operated (1220). The determination may be made when,for example, the vehicle is started, switched into autonomous mode, orwhen the vehicle resets or re-determines its position for a particularreason. In some examples, the AV control system 400 may approximate thecurrent location of the vehicle 1110 using a satellite navigationcomponent and/or historical information (e.g., information from a priortrip).

The AV control system 400 performs localization by determining alocation of the vehicle within the determined submap. In particular, thelocalization may be performed by comparing current sensor data (e.g.,current image data 1043) to a previously determined sensorrepresentation (e.g., sensor state 1029) of the region surrounding thecurrent road segment, where each of the current sensor data and theprior sensor data are associated with a particular location of capture(1230). In an implementation, the selected submap may include orotherwise identify the prior sensor information, as well as provide aprior location of capture within the submap, and the AV control system400 may compare current and past sensor information in order todetermine an accurate and highly granular (e.g., within 1 foot) locationof the vehicle within the submap. In some examples, the determinedlocation may be relative to a boundary or reference location of thesubmap, and further may be based on a known location of capture for theprior sensor information.

In some examples, the submap carries additional determinationspertaining to the prior sensor state 1029, such as the distance of thevehicle from the sidewalk. The additional determinations may providefurther context and mapping with respect to understanding the currentlocation of the vehicle in the submap. For example, the determinedlocation may be highly granular, specifying, for example, the lane thevehicle 1110 occupies, and a distance of the vehicle from, for example,an edge of the roadway. Still further, in some examples, the location ofthe vehicle 10 may be specific to, for example, a width of a tire.

According to some examples, the AV control system 400 compares featuresof the scene surrounding the vehicle 1110, as provided by current sensordata, to features of the prior sensor state 1029, in order to determinea depicted spatial or geometric differential between the feature asdepicted by the current sensor data and the same feature as depicted bythe prior sensor state (1232). For example, the image processingcomponent 1038 may recognize a given feature from current image data(e.g., Lidar image, sonar image, depth image, etc.) but the givenfeature as recognized from the current image data may vary in dimension(e.g., shape, footprint), spacing (e.g., relative to another object),and/or orientation as compared to the depiction of the feature with theprior sensor state data 1029 of the submap. The identified differentialbetween the respective depictions may be correlative to a spatialdifference between the current location of capture for the vehicle 1110and the prior location of capture associated with the sensor informationof the submap. The determined spatial difference may identify a preciselocation of the vehicle within the area of the submap (1234).

As described with examples of FIG. 13 and FIG. 14, the feature set ofthe current submap may be implemented in the form of a point cloudstructure of sensor data sets, where individual features of the currentsubmap are associated with a precise location. The current sensor dataset may be analyzed to determine sensor data sets which match to pointcloud elements of the point cloud. Based on the comparison, the locationof the vehicle may be determined in reference to the precise location ofpoint cloud elements which form the basis of the comparison.

In FIG. 13, the AV control system 400 employs passive image sensors inconnection with prior sensor state information that is structured in theform of a point cloud. A set of current sensor data may be captured bypassive image sensor devices of the vehicle 10 instance when the vehicletraverses a given area of the road network (1310). The passive imagesensor devices may correspond to, for example, one or more pairs ofstereoscopic cameras of an autonomous vehicle 10.

The AV control system 400 may match a subset of the passive image datato one or more features of a known feature set for an area of the roadnetwork (1320). The known feature sets may be in the form of image-basedsensor data sets, such as feature vectors or image signatures of one ormore static objects which are known to be visible in the area of thevehicle's location. The known features, depicted with the image-basedsensor data sets, may be associated with a precise location. While someexamples such as described with an example of FIG. 11 utilize submapdata structures to carry feature sets which provide a basis forcomparison to current sensor state information, in variations, otherdata structure environments may be used by the vehicle to maintain anduse features for comparing passive image data to corresponding imagereference data. For example, the known features may be associated with aprecise distance and orientation with respect to a roadway landmark(e.g., the end of an intersection). In examples in which submaps areused, the known features may be associated with a precise locationwithin the submap.

The AV control system 400 may determine the location of the vehiclewithin the given area based on the comparison of the current sensorstate 493, which may be in the form of passive image data, and the knownfeatures which are structured in a point cloud and associated with aknown reference location (1330). In one implementation, aspects offeatures detected from the current sensor state 493 are compared tocorresponding aspects of the matched and known feature set. One or morevariations are determined with respect to dimension and pose, as betweenthe aspects of the features provided in the current sensor data andcorresponding aspects of the matched feature set (1332). The variationsmay be converted into position and pose variation with respect to thereference location of the reference images.

With regard to FIG. 14, the AV control system 400 may access multiplesets of reference imagelets (e.g., a plurality of point clouds) whichdepict known features of the area of the road network for the vehicle'scurrent location (1410). Each of the reference imagelet sets may depicta feature that is associated with a reference location, identifying, forexample, a location of a camera where the set of imagelets werepreviously captured.

While the vehicle traverses a road segment, the AV control system 400obtains current image data from one or more camera devices of thevehicle (1420). As the vehicle traverses the road network, the AVcontrol system 400 may also associate an approximate location with thecurrent image data. The approximate location may be determined by, forexample, a satellite navigation component and/or historical informationwhich tracks or records the location of the vehicle.

For the given location, the AV control system 400 selects one of themultiple sets of reference imagelets, based at least in part on theapproximate location of the vehicle (1430). As an addition or variation,the vehicle may have alternative point cloud representations to selectfrom for a given reference location, with the alternative point cloudrepresentations representing alternative lighting conditions (e.g.,seasonal, from weather, time of day, etc.).

Additionally, the AV control system 400 makes a determination that anobject depicted by the current image data matches to a feature depictedby one of the imagelets of the matching set (1440). For example,individual imagelets of the selected reference imagelet set may becompared to portions of the current image data in order to determine thepresence of a matching object or object feature. In some examples, theAV control system 400 may utilize rules, models or other logic tooptimize the use of point cloud imagelets for purpose of determininglocation. In one aspect, a set of selection rules may be utilized toidentify imagelets of the known feature set to either use or ignore whenperforming the comparison to the current sensor state. The selectionrules may be based in part on context, such as time of day, weathercondition, and/or lighting conditions. For example, the selection rulesmay disregard imagelets that depict vertical surfaces when there issnow.

In making the determination, the AV control system 400 applies atransformation on one of the current image data or the referenceimagelets of the selected set (1442). The AV control system 400 mayapply the transformation based on a determination that a variation of alighting condition is present as between the area of the road networkwhen the reference set of imagelets were captured and when the currentimage data is captured (1444). The condition may be of a type whichaffects an appearance of objects and/or the surrounding area. In someexamples, the condition may be one that affects a lighting of the areasurrounding the road network, such as the time of day (e.g., variationin image as captured during daytime, dusk or evening) or weather (e.g.,cloudless sky versus heavy inclement weather).

According to some examples, the transformation may be determined from acriteria or model that is trained using sensor data previously collectedfrom the current vehicle and/or one or more other vehicles at theapproximate location (1446). In variations, the transformation may bedetermined from a model that is trained using sensor data previouslycollected from the current vehicle and/or one or more other vehicles ata neighboring location (1448). The neighboring location may be, forexample, on the same road (e.g., on an alternative region of the roadwhere more sensor data exists), on an adjacent road (e.g., on aneighboring road where the condition is deemed to have a similaraffect), or on a same type of road (e.g., road darkened by trees).

The AV control system 400 determines a highly granular location of thevehicle 10 based at least in part on the reference location associatedwith the feature depicted by the current image data (1450). In someexamples, a dimensional or geometric aspect of the object is compared toa corresponding dimension or geometric aspect of the object depicted bythe reference imagelets in order to determine one or more visualdifferentials (1452). In variations, the object depicted by the currentimage data is altered or warped to provide the differential indimension, orientation or other geometric characteristic until theobject depicted by the current image data matches that of the referenceimage (1454). The differential(s) may be mapped or translated into adifference in distance and orientation with respect to the referencelocation of the reference imagelet where the corresponding feature isdepicted.

With reference to FIG. 15, a vehicle is autonomously operated to travelacross the road segment using, for example, AV control system 400,including the AV control system 400. The vehicle 1110 may obtain currentsensor data as the vehicle traverses a road segment (1510). The currentsensor data may include image data, such as captured by two-dimensionalcameras, or by pairs of stereoscopic cameras that capturethree-dimensional images of a corresponding scene. In variations, thesensor data may include Lidar or radar images.

In traversing the road segment, the vehicle 1110 may access storedsensor data which identifies a set of static objects (1520). The set ofstatic objects are identified by the vehicle based on vehicle location.In some implementations, the vehicle may have a granular awareness ofits own location, using for example, a satellite navigation component,or a general determination made from historical data or from aparticular submap in use.

In variations, the stored sensor data that identifies the static objectsmay reside with a submap that identifies the precise location of eachstatic object relative to the submap. For example, the vehicle mayutilize the stored submaps to concurrently perform localization so thatthe vehicle's current location within the road segment is known.Additionally, the submap may identify the location of static objects inrelation to a reference frame of the submap. In such examples, thevehicle may facilitate its ability to identify stored data that depictsthose static objects which are most likely present and depicted by thecurrent sensor state 493 of the vehicle 1110.

The vehicle 1110 may determine one or more non-static (or dynamic)objects as being present in a vicinity of the vehicle based on thecurrent sensor data and the stored sensor data (1530).

In determining one or more non-static objects as being present, the AVcontrol system 400 may reduce a quantity of sensor data that isprocessed based on portions of the stored sensor data that are deemed todepict one or more of the static objects (1532). For example, the AVcontrol system 400 may subtract portions of the current image data whichare deemed to depict any one or more of the set of static objects. Indetermining portions of the current sensor data which depict staticobjects, the AV control system 400 may implement image analyses in orderto recognize or detect the static objects of the stored sensor datawhich are likely depicted by the current sensor data.

According to some examples, once the AV control system 400 determinesthat a non-static object is present in a vicinity of the vehicle, the AVcontrol system 400 tracks the non-static object as the vehicleprogresses on the road segment (1540). Depending on implementation, thevehicle may track the non-static object using any one or combination ofsensors, including cameras, Lidar, radar or sonar.

In some examples, the AV control system 400 may track the non-staticobject without tracking any of the determined static objects (1542). Forexample, the AV control system 400 may identify portions of an overallpixel map which are likely to depict static objects. The AV controlsystem 400 may then ignore portions of the current image data which mapto the identified portions of the pixel map which depict static objects.

The AV control system 400 may track the non-static object by determininga trajectory of the object (1544). The trajectory determination caninclude sampling for the position of the non-static object for a shortduration of time, while ignoring the static objects. The trajectorydetermination may include a predicted trajectory of the object (1546),based on probability or worst-case scenario.

It is contemplated for embodiments described herein to extend toindividual elements and concepts described herein, independently ofother concepts, ideas or system, as well as for embodiments to includecombinations of elements recited anywhere in this application. Althoughembodiments are described in detail herein with reference to theaccompanying drawings, it is to be understood that the invention is notlimited to those precise embodiments. As such, many modifications andvariations will be apparent to practitioners skilled in this art.Accordingly, it is intended that the scope of the invention be definedby the following claims and their equivalents. Furthermore, it iscontemplated that a particular feature described either individually oras part of an embodiment can be combined with other individuallydescribed features, or parts of other embodiments, even if the otherfeatures and embodiments make no mentioned of the particular feature.Thus, the absence of describing combinations should not preclude theinventor from claiming rights to such combinations.

What is claimed is:
 1. A method for performing localization for avehicle in motion, the method being implemented by one or moreprocessors and comprising: obtaining a set of sensor data captured bythe vehicle at a given instance as the vehicle traverses a given area ofa road network, the sensor data including passive image data; matchingan image subset of the passive image data to a feature from a knownfeature set for the given area of the road network; and determining thelocation of the vehicle within the given area based on referencelocation information previously associated with the known feature set.2. The method of claim 1, wherein matching the image subset includesdetecting an object depicted by the image subset as corresponding to thefeature of the known feature set.
 3. The method of claim 2, whereindetermining the location is based on determining a difference between asize of the object depicted by the image subset and a dimensionassociated with the feature when the feature is perceived from a givenlocation.
 4. The method of claim 2, wherein determining the location isbased on determining a difference in perspective between the objectdepicted by the image subset and a perspective associated with thefeature when the feature is perceived from a given location.
 5. Themethod of claim 1, further comprising determining an orientation of thevehicle relative to a direction of travel as the vehicle progressesthrough the given area.
 6. The method of claim 1, wherein matching theimage subset is based on a manipulation of the image subset so that theobject depicted by the image subset more closely matches the feature ofthe known feature set.
 7. The method of claim 1, further comprisingselecting the feature of the known feature set based on one or moreconditions that affect an appearance of individual features of the knownfeature set.
 8. The method of claim 7, wherein the one or moreconditions are based on at least one of weather, season, or time of day.9. The method of claim 8, wherein the one or more conditions include atleast one of (i) a presence of precipitation, or (ii) a presence ofsnow.
 10. The method of claim 7, further comprising identifying ageometry of individual features in the set of features, and whereinselecting the feature is based at least in part on the geometry of thefeature.
 11. The method of claim 10, wherein selecting the featureincludes selecting features that have a vertical geometry when the oneor more conditions correspond to the presence of precipitation or snow.12. The method of claim 1, wherein matching the image subset includesimplementing one or more rules based on the presence of one or moreconditions that affect an appearance of individual features of the knownfeature set.
 13. The method of claim 12, wherein implementing the one ormore rules includes avoiding using white pixels of the passive image setfor the image subset when snow is present in the given area.
 14. Themethod of claim 12, wherein implementing the one or more rules includesavoiding using image data of the passive image set which is below ahorizon of a camera of the vehicle used to capture the passive image setwhen the snow is present in the given area.
 15. The method of claim 1,further comprising: determining a submap from a collection of submapsthat corresponds to the area of the road network where the vehicle islocated, wherein the submap includes one or more data layers whichinclude the known feature set.
 16. The method of claim 15, wherein theone or more data layers provide a three-dimensional cloud of points, inwhich individual points of the cloud are associated with imageletsrepresenting one or more features of the known feature set.
 17. Themethod of claim 15, wherein individual points of the cloud of points areassociated with reference information indicating at least one of alocation or perspective of a sensor device used to capture theassociated imagelet at a previous instance in time.
 18. The method ofclaim 15, further comprising selecting a set of points with associatedimagelets from a larger number of points of the pointcloud withassociated imagelets based on information associated with individualpoints in the larger number of points, the associated imagelets of theselected set of points corresponding to the known feature set, andwherein the information associated with individual points in the largernumber of points include at least one of a semantic label, geometricinformation and/or contextual information.
 19. A computer systemcomprising: a memory to store a set of instructions; one or moreprocessors to use the set of instructions to: obtain a set of sensordata captured by the vehicle at a given instance as the vehicletraverses a given area of a road network, the sensor data includingpassive image data; match an image subset of the passive image data to afeature from a known feature set for the given area of the road network;and determineS the location of the vehicle within the given area basedon reference location information previously associated with the knownfeature set.
 20. The computer system of claim 19, wherein the computersystem is provided on an autonomous vehicle.
 21. The computer system ofclaim 19, wherein the computer system communicates with an autonomousvehicle over a network.